[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:18 dist-test-slave-pkws krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:18 dist-test-slave-pkws krb5kdc[399](info): set up 2 sockets
Sep 27 00:41:18 dist-test-slave-pkws krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 00:41:19.613300 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:19 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:19.736218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:19.845196 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:19.845436 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:19.845474 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:19.849426 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:19.849478 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:19.849496 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:19.849539 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:19.849574 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:19.854835 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:19.855842 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:19.856982 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:19.865806 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:19.865816 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:19.865942 412 server_base.cc:1034] running on GCE node
W20260927 00:41:19.866080 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:19.866576 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:19.867702 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:19.868947 412 hybrid_clock.cc:648] HybridClock initialized: now 1790469679868834 us; error 115 us; skew 500 ppm
Sep 27 00:41:19 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:19.871812 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:19.872696 412 webserver.cc:460] Webserver started at http://127.0.98.62:38541/ using document root <none> and password file <none>
I20260927 00:41:19.873334 412 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:19.873423 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:19.873659 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:19.875677 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "e285d08139aa43fcafb6767a1c4951ff"
format_stamp: "Formatted at 2026-09-27 00:41:19 on dist-test-slave-pkws"
server_key: "50b40465e775366ec85921a03715f04e"
server_key_iv: "94ca34af0ca6144a5bf88e8ab626b3c0"
server_key_version: "encryptionkey@0"
I20260927 00:41:19.876241 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "e285d08139aa43fcafb6767a1c4951ff"
format_stamp: "Formatted at 2026-09-27 00:41:19 on dist-test-slave-pkws"
server_key: "50b40465e775366ec85921a03715f04e"
server_key_iv: "94ca34af0ca6144a5bf88e8ab626b3c0"
server_key_version: "encryptionkey@0"
I20260927 00:41:19.880025 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:19.882505 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:19.883653 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:19.883790 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "e285d08139aa43fcafb6767a1c4951ff"
format_stamp: "Formatted at 2026-09-27 00:41:19 on dist-test-slave-pkws"
server_key: "50b40465e775366ec85921a03715f04e"
server_key_iv: "94ca34af0ca6144a5bf88e8ab626b3c0"
server_key_version: "encryptionkey@0"
I20260927 00:41:19.883900 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:19.910881 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:19.913949 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:19.914181 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:19.921461 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44317
I20260927 00:41:19.921479 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44317 every 8 connection(s)
I20260927 00:41:19.922415 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:19.925209 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:19.930884 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: Bootstrap starting.
I20260927 00:41:19.932452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 412
I20260927 00:41:19.932618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:19.933995 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:19.934398 392 external_mini_cluster.cc:1428] Setting key 7a9e2e4fcd5f1c44e2730b8a1d3fda64
I20260927 00:41:19.934967 480 log.cc:824] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:19.937271 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: No bootstrap required, opened a new log
I20260927 00:41:19.942886 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260927 00:41:19.943171 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:19.943248 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e285d08139aa43fcafb6767a1c4951ff, State: Initialized, Role: FOLLOWER
I20260927 00:41:19.943769 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260927 00:41:19.943935 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:19.944010 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:19.944185 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:19.945202 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260927 00:41:19.945534 480 leader_election.cc:302] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e285d08139aa43fcafb6767a1c4951ff; no voters:
I20260927 00:41:19.945809 480 leader_election.cc:288] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:19.945921 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:19.946130 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [term 1 LEADER]: Becoming Leader. State: Replica: e285d08139aa43fcafb6767a1c4951ff, State: Running, Role: LEADER
I20260927 00:41:19.946540 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260927 00:41:19.947062 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:19.949918 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e285d08139aa43fcafb6767a1c4951ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } } }
I20260927 00:41:19.949954 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e285d08139aa43fcafb6767a1c4951ff. Latest consensus state: current_term: 1 leader_uuid: "e285d08139aa43fcafb6767a1c4951ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e285d08139aa43fcafb6767a1c4951ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } } }
I20260927 00:41:19.950057 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:19.950068 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff [sys.catalog]: This master's current role is: LEADER
W20260927 00:41:19.952399 499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:19.952484 499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:19.952579 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:19.953583 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 00:41:19 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:19.959667 501 catalog_manager.cc:1347] Generated new cluster ID: aaa87fdee813463b8ee28737cf969f92
I20260927 00:41:19.959792 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:19.962893 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:19.937917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.0.98.62:44317)
0927 00:41:19.938395 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:19.938416 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:19.939637 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 00:41:19.943570 (+ 3933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:19.943590 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:19.943960 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:19.944338 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:19.947051 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:19.948741 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:19.950569 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:19.951205 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:19.951368 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:19.956662 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:19.956669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:19.956685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:19.956716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:19.958952 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:19.959635 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:19.959642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:19.959647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:19.959831 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:19.960319 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:19.960323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:19.960325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:19.960836 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:19.961081 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:19.961721 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 00:41:19.962013 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":130,"threads_started":1}
I20260927 00:41:19.978173 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:19.979398 501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:19.987720 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e285d08139aa43fcafb6767a1c4951ff: Generated new TSK 0
I20260927 00:41:19.988572 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:20.020596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:20.126487 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:20.126722 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:20.126765 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:20.131359 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:20.131413 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:20.131510 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:20.136675 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:20.137652 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:20.138617 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:20.147979 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:20.147996 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:20.148027 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:20.148216 506 server_base.cc:1034] running on GCE node
I20260927 00:41:20.148741 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:20.149462 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:20.150645 506 hybrid_clock.cc:648] HybridClock initialized: now 1790469680150634 us; error 35 us; skew 500 ppm
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:20.153287 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:20.154074 506 webserver.cc:460] Webserver started at http://127.0.98.1:40967/ using document root <none> and password file <none>
I20260927 00:41:20.154609 506 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:20.154698 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:20.154915 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:20.156872 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "c32a1b883e7747b68ea4a04a8f038860"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "f590e42d38b6ebf445c10596a23ab826"
server_key_iv: "16cae98218182399775ffa77f10f4525"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.157500 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c32a1b883e7747b68ea4a04a8f038860"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "f590e42d38b6ebf445c10596a23ab826"
server_key_iv: "16cae98218182399775ffa77f10f4525"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.161036 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:41:20.163244 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.164373 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:20.164506 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "c32a1b883e7747b68ea4a04a8f038860"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "f590e42d38b6ebf445c10596a23ab826"
server_key_iv: "16cae98218182399775ffa77f10f4525"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.164610 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:20.181103 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:20.184206 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:20.184417 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:20.185035 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:20.185889 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:20.185956 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.186019 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:20.186072 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.195137 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37509
I20260927 00:41:20.195168 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37509 every 8 connection(s)
I20260927 00:41:20.196018 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:20.206177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 506
I20260927 00:41:20.206306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:20.206557 392 external_mini_cluster.cc:1428] Setting key dfbace07129cc1de6feb2fbc8810920c
I20260927 00:41:20.209709 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.197870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48437 (local address 127.0.98.62:44317)
0927 00:41:20.198012 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:20.198016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:20.198818 (+ 802us) server_negotiation.cc:407] Connection header received
0927 00:41:20.200138 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:20.200142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:20.200199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:20.200276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:20.201455 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.201986 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.203135 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.203363 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.203471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.205843 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:20.205846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:20.205849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:20.205880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:20.207527 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.208120 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.208124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.208125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.208211 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.208543 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.208547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.208548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.208791 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:20.208916 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:20.209389 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 00:41:20.209534 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:41:20.210760 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.198231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44317 (local address 127.0.98.1:48437)
0927 00:41:20.198674 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:20.198716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:20.199927 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:20.200438 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:20.200445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:20.200787 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:20.201307 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.201320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.202111 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.202116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.203000 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.203009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.203517 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.203521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.203692 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.204213 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:20.204229 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:20.205681 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:20.207644 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.207649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.207664 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.208019 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.208327 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.208329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.208330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.208453 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.208934 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:20.208940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:20.209162 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:41:20.209314 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:41:20.209702 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260927 00:41:20.212002 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44317
I20260927 00:41:20.212249 635 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:20.212693 635 heartbeater.cc:511] Master 127.0.98.62:44317 requested a full tablet report, sending...
I20260927 00:41:20.214171 444 ts_manager.cc:194] Registered new tserver with Master: c32a1b883e7747b68ea4a04a8f038860 (127.0.98.1:37509)
I20260927 00:41:20.215260 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48437
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:20.232414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:20.341094 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:20.341331 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:20.341405 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:20.345884 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:20.345942 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:20.346032 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:20.351444 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:20.352437 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:20.353449 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:20.362802 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:20.362838 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:20.362802 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:20.363121 640 server_base.cc:1034] running on GCE node
I20260927 00:41:20.363454 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:20.364046 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:20.365203 640 hybrid_clock.cc:648] HybridClock initialized: now 1790469680365174 us; error 41 us; skew 500 ppm
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:20.367996 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:20.368933 640 webserver.cc:460] Webserver started at http://127.0.98.2:41335/ using document root <none> and password file <none>
I20260927 00:41:20.369446 640 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:20.369503 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:20.369661 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:20.371537 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "3583612664a34263bff4ce75542ea8c1"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "75f2287b7581d72a7aa8e945b0d9c641"
server_key_iv: "b20ef6459a8b446d81f70618479d505b"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.372022 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3583612664a34263bff4ce75542ea8c1"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "75f2287b7581d72a7aa8e945b0d9c641"
server_key_iv: "b20ef6459a8b446d81f70618479d505b"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.375923 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:41:20.378276 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.379431 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:20.379573 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "3583612664a34263bff4ce75542ea8c1"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "75f2287b7581d72a7aa8e945b0d9c641"
server_key_iv: "b20ef6459a8b446d81f70618479d505b"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.379683 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:20.390439 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:20.393570 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:20.393769 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:20.394320 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:20.395089 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:20.395160 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.395228 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:20.395277 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.404621 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46809
I20260927 00:41:20.404641 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46809 every 8 connection(s)
I20260927 00:41:20.405510 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:20.407770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 640
I20260927 00:41:20.407891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:20.408129 392 external_mini_cluster.cc:1428] Setting key 5fd802515fabfd005082c36f9af3ec6b
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:20.419625 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.407274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37727 (local address 127.0.98.62:44317)
0927 00:41:20.407402 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:20.407406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:20.408288 (+ 882us) server_negotiation.cc:407] Connection header received
0927 00:41:20.410279 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:20.410282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:20.410331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:20.410404 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:20.411506 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.412034 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.413221 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.413442 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.413554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.415919 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:20.415922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:20.415924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:20.415946 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:20.417464 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.417986 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.417994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.417995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.418065 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.418416 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.418421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.418423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.418599 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:20.418704 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:20.419277 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 00:41:20.419391 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:41:20.420365 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.407644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44317 (local address 127.0.98.2:37727)
0927 00:41:20.408119 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:20.408167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:20.410075 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:20.410564 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:20.410571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:20.410912 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:20.411364 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.411377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.412161 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.412166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.413098 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.413107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.413584 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.413589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.413764 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.414345 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:20.414360 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:20.415754 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:20.417588 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.417594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.417608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.417853 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.418164 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.418168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.418170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.418319 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.418703 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:20.418709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:20.419019 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 00:41:20.419227 (+ 208us) client_negotiation.cc:240] Negotiation successful
0927 00:41:20.419549 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":126,"threads_started":1}
I20260927 00:41:20.421526 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44317
I20260927 00:41:20.421793 769 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:20.422381 769 heartbeater.cc:511] Master 127.0.98.62:44317 requested a full tablet report, sending...
I20260927 00:41:20.423437 444 ts_manager.cc:194] Registered new tserver with Master: 3583612664a34263bff4ce75542ea8c1 (127.0.98.2:46809)
I20260927 00:41:20.423987 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:20.436086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:20.543653 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:20.543892 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:20.543933 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:20.548100 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:20.548153 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:20.548256 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:20.553478 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44317
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:20.554414 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:20.555393 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:20.564860 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:20.564937 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:20.564931 774 server_base.cc:1034] running on GCE node
W20260927 00:41:20.564868 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:20.565486 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:20.566111 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:20.567306 774 hybrid_clock.cc:648] HybridClock initialized: now 1790469680567272 us; error 50 us; skew 500 ppm
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:20.570152 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:20.570945 774 webserver.cc:460] Webserver started at http://127.0.98.3:38431/ using document root <none> and password file <none>
I20260927 00:41:20.571496 774 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:20.571589 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:20.571805 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:20.573832 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "46eb100b888f49c2bb905dfaa0530502"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "9b09e0975de905c55dc46143d6f20c96"
server_key_iv: "19a7991608edd8a8c7b1d7e75f6c63ce"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.574380 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "46eb100b888f49c2bb905dfaa0530502"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "9b09e0975de905c55dc46143d6f20c96"
server_key_iv: "19a7991608edd8a8c7b1d7e75f6c63ce"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.578307 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:41:20.580633 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.581895 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:20.582031 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "46eb100b888f49c2bb905dfaa0530502"
format_stamp: "Formatted at 2026-09-27 00:41:20 on dist-test-slave-pkws"
server_key: "9b09e0975de905c55dc46143d6f20c96"
server_key_iv: "19a7991608edd8a8c7b1d7e75f6c63ce"
server_key_version: "encryptionkey@0"
I20260927 00:41:20.582139 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:20.600684 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:20.603420 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:20.603626 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:20.604269 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:20.605131 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:20.605202 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.605269 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:20.605319 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:20.613924 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37397
I20260927 00:41:20.613960 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37397 every 8 connection(s)
I20260927 00:41:20.614763 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:20.621712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 774
I20260927 00:41:20.621831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:20.622092 392 external_mini_cluster.cc:1428] Setting key b123cabd77c32fef77ee4b69fcd826bc
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:20.627960 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.616412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55627 (local address 127.0.98.62:44317)
0927 00:41:20.616540 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:20.616543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:20.617319 (+ 776us) server_negotiation.cc:407] Connection header received
0927 00:41:20.618687 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:20.618691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:20.618750 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:20.618856 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:20.619983 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.620504 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.621688 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.622003 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.622121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.624819 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:20.624822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:20.624824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:20.624847 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:20.626339 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.626778 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.626780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.626782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.626852 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.627121 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.627123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.627124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.627281 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:20.627353 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:20.627763 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 00:41:20.627849 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:20.628738 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.616710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44317 (local address 127.0.98.3:55627)
0927 00:41:20.617171 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:20.617212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:20.618465 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:20.619024 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:20.619030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:20.619371 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:20.619838 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.619851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.620632 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.620635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.621547 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:20.621564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.622152 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.622156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:20.622321 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.622948 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:20.622963 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:20.624657 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:20.626437 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.626441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.626453 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.626692 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.626925 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:20.626927 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:20.626928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:20.627047 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:20.627362 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:20.627367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:20.627569 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 00:41:20.627718 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 00:41:20.627943 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":149,"threads_started":1}
I20260927 00:41:20.629801 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44317
I20260927 00:41:20.630039 903 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:20.630508 903 heartbeater.cc:511] Master 127.0.98.62:44317 requested a full tablet report, sending...
I20260927 00:41:20.631433 444 ts_manager.cc:194] Registered new tserver with Master: 46eb100b888f49c2bb905dfaa0530502 (127.0.98.3:37397)
I20260927 00:41:20.631986 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55627
I20260927 00:41:20.636127 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:20.660465 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.651842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.1:37509)
0927 00:41:20.652210 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:20.652219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:20.652240 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:20.652287 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:20.652291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:20.652462 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:20.652578 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:20.653256 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.654020 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.654794 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.655134 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.655214 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.657523 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:20.657531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:20.657543 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:20.657565 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:20.658973 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.659344 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.659348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.659350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.659424 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.659673 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.659675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.659676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.659888 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:20.660038 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:20.660198 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:41:20.660319 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":153,"threads_started":1}
W20260927 00:41:20.661425 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60540
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:20.686151 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:20.677982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.98.1:37509)
0927 00:41:20.678134 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:20.678139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:20.678292 (+ 153us) server_negotiation.cc:407] Connection header received
0927 00:41:20.678449 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:20.678458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:20.678498 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:20.678579 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:20.679252 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.679780 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.680602 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:20.680836 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:20.680987 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:20.683242 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:20.683251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:20.683253 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:20.683274 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:20.684475 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.684920 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.684925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.684927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.685007 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:20.685283 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:20.685288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:20.685290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:20.685450 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:20.685594 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:20.685871 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:41:20.685987 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:41:20.687144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 506
I20260927 00:41:20.692853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 640
I20260927 00:41:20.699379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 774
I20260927 00:41:20.703810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 412
2026-09-27T00:41:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2135 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[917](info): set up 2 sockets
Sep 27 00:41:20 dist-test-slave-pkws krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 00:41:21.728467 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:21 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:21.854091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:21.959277 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:21.959560 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:21.959630 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:21.963649 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:21.963747 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:21.963788 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:21.963850 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:21.963891 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:21.968997 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:21.970039 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:21.971048 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:21.979266 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:21.979238 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:21.979475 930 server_base.cc:1034] running on GCE node
W20260927 00:41:21.979481 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:21.980124 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:21.981213 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:21.982411 930 hybrid_clock.cc:648] HybridClock initialized: now 1790469681982399 us; error 26 us; skew 500 ppm
Sep 27 00:41:21 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:21.985082 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:21.985945 930 webserver.cc:460] Webserver started at http://127.0.98.62:44091/ using document root <none> and password file <none>
I20260927 00:41:21.986476 930 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:21.986564 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:21.986788 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:21.988737 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "223150577802496eae7a89758d9b4398"
format_stamp: "Formatted at 2026-09-27 00:41:21 on dist-test-slave-pkws"
server_key: "6365696430c81c148752bb497b7eb19f"
server_key_iv: "3eedd70c774bf7007a7df022f19a1143"
server_key_version: "encryptionkey@0"
I20260927 00:41:21.989358 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "223150577802496eae7a89758d9b4398"
format_stamp: "Formatted at 2026-09-27 00:41:21 on dist-test-slave-pkws"
server_key: "6365696430c81c148752bb497b7eb19f"
server_key_iv: "3eedd70c774bf7007a7df022f19a1143"
server_key_version: "encryptionkey@0"
I20260927 00:41:21.992792 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:21.995065 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:21.996177 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:21.996318 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "223150577802496eae7a89758d9b4398"
format_stamp: "Formatted at 2026-09-27 00:41:21 on dist-test-slave-pkws"
server_key: "6365696430c81c148752bb497b7eb19f"
server_key_iv: "3eedd70c774bf7007a7df022f19a1143"
server_key_version: "encryptionkey@0"
I20260927 00:41:21.996423 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:22.015321 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:22.018175 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:22.018374 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:22.025368 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45345
I20260927 00:41:22.025400 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45345 every 8 connection(s)
I20260927 00:41:22.026293 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:22.029088 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:22.029675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 930
I20260927 00:41:22.029775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:22.030051 392 external_mini_cluster.cc:1428] Setting key 494f434e1ae2363ead78916351549bb5
I20260927 00:41:22.035765 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398: Bootstrap starting.
I20260927 00:41:22.037961 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398: Neither blocks nor log segments found. Creating new log.
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:22.038708 998 log.cc:824] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:22.041163 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398: No bootstrap required, opened a new log
I20260927 00:41:22.044123 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.031319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.98.62:45345)
0927 00:41:22.031704 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.031718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.031759 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:41:22.032413 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.032429 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.032735 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.033109 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.033956 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.034915 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.035773 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.036284 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.036450 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.038993 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.038999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.039017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.039054 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.041045 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.041509 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.041518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.041525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.041659 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.041910 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.041913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.041915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.042408 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.042595 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.042796 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.043088 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":133,"threads_started":1}
I20260927 00:41:22.045804 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } }
I20260927 00:41:22.046072 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:22.046126 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223150577802496eae7a89758d9b4398, State: Initialized, Role: FOLLOWER
I20260927 00:41:22.046564 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } }
I20260927 00:41:22.046723 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:22.046799 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:22.046911 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:22.047794 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } }
I20260927 00:41:22.048144 998 leader_election.cc:302] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 223150577802496eae7a89758d9b4398; no voters:
I20260927 00:41:22.048481 998 leader_election.cc:288] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:22.048584 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:22.048774 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [term 1 LEADER]: Becoming Leader. State: Replica: 223150577802496eae7a89758d9b4398, State: Running, Role: LEADER
I20260927 00:41:22.049109 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } }
I20260927 00:41:22.049739 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:22.050792 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "223150577802496eae7a89758d9b4398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } } }
I20260927 00:41:22.050909 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:22.051294 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:22.051275 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 223150577802496eae7a89758d9b4398. Latest consensus state: current_term: 1 leader_uuid: "223150577802496eae7a89758d9b4398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223150577802496eae7a89758d9b4398" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45345 } } }
I20260927 00:41:22.051410 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:22.054744 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:22.060034 1012 catalog_manager.cc:1347] Generated new cluster ID: 919d39f4bbf14e6885c7f349c26c8699
I20260927 00:41:22.060132 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:22.070945 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:22.071871 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:22.075356 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 223150577802496eae7a89758d9b4398: Generated new TSK 0
I20260927 00:41:22.075973 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:22.112154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:22.234457 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:22.234712 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:22.234789 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:22.238768 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:22.238842 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:22.238926 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:22.243921 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:22.244848 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:22.245795 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:22.255054 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:22.255059 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:22.255124 1024 server_base.cc:1034] running on GCE node
W20260927 00:41:22.255064 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:22.255664 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:22.256275 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:22.257483 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790469682257444 us; error 31 us; skew 500 ppm
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:22.260062 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:22.260870 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43985/ using document root <none> and password file <none>
I20260927 00:41:22.261436 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:22.261523 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:22.261742 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:22.263602 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "f27430433eef4bb393af93cece29f310"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "cea276355ee22ae51502bcd6085d9aae"
server_key_iv: "934665ef24996ca0276bbdc45dc34689"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.264154 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f27430433eef4bb393af93cece29f310"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "cea276355ee22ae51502bcd6085d9aae"
server_key_iv: "934665ef24996ca0276bbdc45dc34689"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.267630 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:41:22.270020 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.271258 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:22.271409 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f27430433eef4bb393af93cece29f310"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "cea276355ee22ae51502bcd6085d9aae"
server_key_iv: "934665ef24996ca0276bbdc45dc34689"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.271520 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:22.283691 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:22.286363 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:22.286542 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:22.287065 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:22.287823 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:22.287892 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.287947 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:22.287976 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.296787 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34091
I20260927 00:41:22.296805 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34091 every 8 connection(s)
I20260927 00:41:22.297616 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:22.298242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1024
I20260927 00:41:22.298354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:22.298609 392 external_mini_cluster.cc:1428] Setting key e4885c1f74c800cf3f2896fc2277b084
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:22.313000 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.299524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51459 (local address 127.0.98.62:45345)
0927 00:41:22.299670 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.299673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.300671 (+ 998us) server_negotiation.cc:407] Connection header received
0927 00:41:22.302463 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.302467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.302526 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.302624 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.304016 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.304538 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.306363 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.306621 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.306703 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.309395 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.309398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.309400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.309422 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.310908 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.311512 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.311516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.311517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.311595 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.311944 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.311948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.311950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.312122 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.312223 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.312690 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.312807 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:22.313743 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.299991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45345 (local address 127.0.98.1:51459)
0927 00:41:22.300515 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.300570 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.302259 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.302768 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.302774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.303204 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:22.303865 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.303886 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.304665 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.304669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.306238 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.306252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.306792 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.306796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.306999 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.307584 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:22.307607 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:22.309231 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:22.311087 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.311095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.311109 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.311413 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.311685 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.311689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.311691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.311844 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.312232 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:22.312241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:22.312463 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.312637 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.312971 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260927 00:41:22.314976 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45345
I20260927 00:41:22.315231 1153 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:22.315819 1153 heartbeater.cc:511] Master 127.0.98.62:45345 requested a full tablet report, sending...
I20260927 00:41:22.317343 962 ts_manager.cc:194] Registered new tserver with Master: f27430433eef4bb393af93cece29f310 (127.0.98.1:34091)
I20260927 00:41:22.318432 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:22.325960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:22.434149 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:22.434398 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:22.434468 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:22.438385 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:22.438439 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:22.438534 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:22.443688 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:22.444696 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:22.445727 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:22.455250 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:22.455250 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:22.455260 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:22.455440 1158 server_base.cc:1034] running on GCE node
I20260927 00:41:22.455842 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:22.456426 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:22.457607 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790469682457587 us; error 31 us; skew 500 ppm
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:22.460210 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:22.461364 1158 webserver.cc:460] Webserver started at http://127.0.98.2:44243/ using document root <none> and password file <none>
I20260927 00:41:22.461859 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:22.461916 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:22.462066 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:22.464018 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "3fe3b0109610481083fce340eeb5d641"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "57b01db39be09caa246b209ee3c212a6"
server_key_iv: "e0dee5427fabad04e519e5acbc62128a"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.464545 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3fe3b0109610481083fce340eeb5d641"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "57b01db39be09caa246b209ee3c212a6"
server_key_iv: "e0dee5427fabad04e519e5acbc62128a"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.468068 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:22.470391 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.471560 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:22.471648 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "3fe3b0109610481083fce340eeb5d641"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "57b01db39be09caa246b209ee3c212a6"
server_key_iv: "e0dee5427fabad04e519e5acbc62128a"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.471724 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:22.480619 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:22.483299 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:22.483434 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:22.483892 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:22.484570 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:22.484615 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.484651 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:22.484673 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.492998 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45085
I20260927 00:41:22.493008 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45085 every 8 connection(s)
I20260927 00:41:22.493846 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:22.501539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1158
I20260927 00:41:22.501672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:22.501922 392 external_mini_cluster.cc:1428] Setting key 7d9a3799b1cab6800e410ab4c9e8388c
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:22.507963 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.495538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35083 (local address 127.0.98.62:45345)
0927 00:41:22.495680 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.495684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.496438 (+ 754us) server_negotiation.cc:407] Connection header received
0927 00:41:22.497949 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.497953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.498010 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.498116 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.499202 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.499904 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.501179 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.501413 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.501525 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.504172 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.504178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.504185 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.504207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.505660 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.506269 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.506272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.506273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.506344 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.506730 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.506732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.506733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.506901 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.506998 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.507600 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.507720 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:41:22.508656 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.495849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45345 (local address 127.0.98.2:35083)
0927 00:41:22.496290 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.496330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.497734 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.498284 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.498290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.498624 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:22.499059 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.499071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.500023 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.500026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.501055 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.501064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.501558 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.501561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.501758 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.502363 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:22.502381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:22.504014 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:22.505781 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.505789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.505806 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.506171 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.506433 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.506439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.506444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.506637 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.507012 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:22.507018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:22.507345 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.507550 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.507882 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260927 00:41:22.509804 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45345
I20260927 00:41:22.510056 1287 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:22.510607 1287 heartbeater.cc:511] Master 127.0.98.62:45345 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:22.511754 962 ts_manager.cc:194] Registered new tserver with Master: 3fe3b0109610481083fce340eeb5d641 (127.0.98.2:45085)
I20260927 00:41:22.512356 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:22.528239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:22.634768 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:22.634991 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:22.635051 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:22.638857 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:22.638902 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:22.638957 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:22.643844 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:22.644735 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:22.645705 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:22.655180 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:22.655200 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:22.655267 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:22.655345 1292 server_base.cc:1034] running on GCE node
I20260927 00:41:22.655784 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:22.656365 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:22.657521 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790469682657489 us; error 43 us; skew 500 ppm
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:22.660111 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:22.660848 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37231/ using document root <none> and password file <none>
I20260927 00:41:22.661388 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:22.661444 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:22.661595 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:22.663422 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "ae3f14c649524badaa06ea44a554ccc4"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "6da3edb0ebb584443009b10527b04a96"
server_key_iv: "5f175d7063570a0ab3b6f71a148699a5"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.663900 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ae3f14c649524badaa06ea44a554ccc4"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "6da3edb0ebb584443009b10527b04a96"
server_key_iv: "5f175d7063570a0ab3b6f71a148699a5"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.667258 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:22.669450 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.670652 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:22.670778 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "ae3f14c649524badaa06ea44a554ccc4"
format_stamp: "Formatted at 2026-09-27 00:41:22 on dist-test-slave-pkws"
server_key: "6da3edb0ebb584443009b10527b04a96"
server_key_iv: "5f175d7063570a0ab3b6f71a148699a5"
server_key_version: "encryptionkey@0"
I20260927 00:41:22.670892 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:22.686169 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:22.688691 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:22.688874 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:22.689452 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:22.690196 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:22.690271 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.690347 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:22.690383 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:22.698673 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37251
I20260927 00:41:22.698706 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37251 every 8 connection(s)
I20260927 00:41:22.699592 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:22.703486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1292
I20260927 00:41:22.703610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:22.703877 392 external_mini_cluster.cc:1428] Setting key 4789c79ac19fae6e1a239b2f0d9a60bc
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:22.713343 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.701385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34277 (local address 127.0.98.62:45345)
0927 00:41:22.701520 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.701523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.702320 (+ 797us) server_negotiation.cc:407] Connection header received
0927 00:41:22.703866 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.703871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.703928 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.704004 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.705675 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.706338 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.707457 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.707658 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.707730 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.710101 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.710104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.710106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.710129 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.711551 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.711982 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.711984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.711985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.712054 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.712340 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.712342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.712343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.712494 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.712564 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.713140 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.713229 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:41:22.714285 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.701716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45345 (local address 127.0.98.3:34277)
0927 00:41:22.702156 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.702199 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.703569 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.704200 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.704209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.704775 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:22.705530 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.705551 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.706468 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.706471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.707354 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.707363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.707786 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.707789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.707934 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.708435 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:22.708450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:22.709948 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:22.711653 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.711657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.711667 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.711898 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.712134 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:22.712136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:22.712138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:22.712256 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:22.712612 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:22.712617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:22.712842 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.713100 (+ 258us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.713363 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260927 00:41:22.715369 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45345
I20260927 00:41:22.715603 1421 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:22.716058 1421 heartbeater.cc:511] Master 127.0.98.62:45345 requested a full tablet report, sending...
I20260927 00:41:22.717195 962 ts_manager.cc:194] Registered new tserver with Master: ae3f14c649524badaa06ea44a554ccc4 (127.0.98.3:37251)
I20260927 00:41:22.717777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:22.717984 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34277
I20260927 00:41:22.728565 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.721438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.0.98.62:45345)
0927 00:41:22.721573 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.721579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.721760 (+ 181us) server_negotiation.cc:407] Connection header received
0927 00:41:22.721886 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.721890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.721948 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.722033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.722677 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.723170 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.723985 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.724199 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.724307 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.725671 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.725675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.725678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.725707 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.726963 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.727368 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.727372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.727374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.727455 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.727719 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.727724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.727725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.727937 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.728076 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.728299 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.728406 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:41:22.733116 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:22.736212 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:22.750458 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.745017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45085 (local address 127.0.0.1:57328)
0927 00:41:22.745313 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.745327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.745409 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.746201 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.746204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.746221 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:22.746328 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.746337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747485 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.747491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.748905 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.748916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.749859 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.749862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.749977 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750014 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.750147 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.750266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":30,"thread_start_us":87,"threads_started":1}
I20260927 00:41:22.750424 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.745198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.0.98.2:45085)
0927 00:41:22.745765 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.745777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.745809 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:41:22.745871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.745875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.746045 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.746472 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:22.746651 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747352 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.749048 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.749739 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.749861 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750022 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.750174 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.750263 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":148,"threads_started":1}
I20260927 00:41:22.750857 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.744481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.98.1:34091)
0927 00:41:22.745474 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.745486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.745510 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:41:22.745586 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.745589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.745814 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.746334 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:22.746452 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747622 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.749065 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.749933 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.750445 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750578 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.750627 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.750720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":96,"threads_started":1}
I20260927 00:41:22.750983 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.745356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.98.3:37251)
0927 00:41:22.745765 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.745777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.745798 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:22.745870 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.745875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.746045 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.746194 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:22.746594 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747674 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.749101 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.749933 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.750050 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750180 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.750719 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.750820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":100,"threads_started":1}
I20260927 00:41:22.750633 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.744430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37251 (local address 127.0.0.1:35912)
0927 00:41:22.745124 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.745139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.745225 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.746350 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.746353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.746371 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:22.746471 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.746477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747819 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.747821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.749003 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.749016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.750237 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.750242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.750318 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750377 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.750442 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.750502 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":53,"thread_start_us":88,"threads_started":1}
I20260927 00:41:22.750985 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.744340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34091 (local address 127.0.0.1:39364)
0927 00:41:22.744808 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.744852 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.745004 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.746003 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.746011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.746035 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:22.746181 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.746190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.747764 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.747767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.748966 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.748976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.750072 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.750076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.750163 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.750184 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.750236 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.750317 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":32,"thread_start_us":66,"threads_started":1}
I20260927 00:41:22.753777 1221 tablet_service.cc:1467] Processing CreateTablet for tablet dc8322ea159f4cac803d41320e1f6c17 (DEFAULT_TABLE table=test-table [id=692453b7eeb34f2991a95bee98b5a328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:22.753877 1355 tablet_service.cc:1467] Processing CreateTablet for tablet dc8322ea159f4cac803d41320e1f6c17 (DEFAULT_TABLE table=test-table [id=692453b7eeb34f2991a95bee98b5a328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:22.754774 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8322ea159f4cac803d41320e1f6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:22.754879 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8322ea159f4cac803d41320e1f6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:22.754709 1087 tablet_service.cc:1467] Processing CreateTablet for tablet dc8322ea159f4cac803d41320e1f6c17 (DEFAULT_TABLE table=test-table [id=692453b7eeb34f2991a95bee98b5a328]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:22.755606 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc8322ea159f4cac803d41320e1f6c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:22.760753 1438 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: Bootstrap starting.
I20260927 00:41:22.761976 1440 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Bootstrap starting.
I20260927 00:41:22.762465 1439 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: Bootstrap starting.
I20260927 00:41:22.762960 1438 tablet_bootstrap.cc:654] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:22.764099 1440 tablet_bootstrap.cc:654] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:22.764123 1439 tablet_bootstrap.cc:654] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:22.764230 1438 log.cc:824] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:22.765345 1440 log.cc:824] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:22.765344 1439 log.cc:824] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:22.766538 1438 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: No bootstrap required, opened a new log
I20260927 00:41:22.766722 1438 ts_tablet_manager.cc:1397] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:22.767095 1440 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: No bootstrap required, opened a new log
I20260927 00:41:22.767246 1439 tablet_bootstrap.cc:492] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: No bootstrap required, opened a new log
I20260927 00:41:22.767282 1440 ts_tablet_manager.cc:1397] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:41:22.767414 1439 ts_tablet_manager.cc:1397] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:41:22.770104 1440 raft_consensus.cc:348] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.770354 1440 raft_consensus.cc:374] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:22.770460 1440 raft_consensus.cc:729] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27430433eef4bb393af93cece29f310, State: Initialized, Role: FOLLOWER
I20260927 00:41:22.770959 1440 consensus_queue.cc:260] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.770888 1438 raft_consensus.cc:348] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.771126 1438 raft_consensus.cc:374] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:22.771212 1438 raft_consensus.cc:729] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe3b0109610481083fce340eeb5d641, State: Initialized, Role: FOLLOWER
I20260927 00:41:22.771489 1439 raft_consensus.cc:348] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.771626 1438 consensus_queue.cc:260] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.771730 1439 raft_consensus.cc:374] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:22.771780 1439 raft_consensus.cc:729] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3f14c649524badaa06ea44a554ccc4, State: Initialized, Role: FOLLOWER
I20260927 00:41:22.771947 1440 ts_tablet_manager.cc:1428] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:41:22.772212 1439 consensus_queue.cc:260] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.772310 1153 heartbeater.cc:503] Master 127.0.98.62:45345 was elected leader, sending a full tablet report...
I20260927 00:41:22.772812 1421 heartbeater.cc:503] Master 127.0.98.62:45345 was elected leader, sending a full tablet report...
I20260927 00:41:22.773540 1438 ts_tablet_manager.cc:1428] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:41:22.773878 1287 heartbeater.cc:503] Master 127.0.98.62:45345 was elected leader, sending a full tablet report...
I20260927 00:41:22.774000 1439 ts_tablet_manager.cc:1428] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260927 00:41:22.791077 1445 raft_consensus.cc:482] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:22.791272 1445 raft_consensus.cc:504] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.792351 1445 leader_election.cc:288] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f27430433eef4bb393af93cece29f310 (127.0.98.1:34091), ae3f14c649524badaa06ea44a554ccc4 (127.0.98.3:37251)
I20260927 00:41:22.795889 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.792604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51237 (local address 127.0.98.1:34091)
0927 00:41:22.792719 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.792723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.792852 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:41:22.792934 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.792937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.792990 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.793078 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:22.793323 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.793869 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.794931 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.795353 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.795477 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.795534 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.795706 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.795767 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:41:22.795889 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.792516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34091 (local address 127.0.98.2:51237)
0927 00:41:22.792711 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.792728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.792814 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.793089 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.793092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.793108 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:22.793222 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.793227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.793982 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.793986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.794813 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.794822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.795476 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.795479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.795583 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.795607 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.795693 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.795761 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":21}
I20260927 00:41:22.796047 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.792757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58807 (local address 127.0.98.3:37251)
0927 00:41:22.792873 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.792878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.793031 (+ 153us) server_negotiation.cc:407] Connection header received
0927 00:41:22.793177 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.793181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.793226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.793308 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:22.793567 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.794043 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.795114 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.795514 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.795628 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.795684 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.795874 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.795939 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:22.796064 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.792590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37251 (local address 127.0.98.2:58807)
0927 00:41:22.792898 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:22.792957 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:22.793072 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:22.793320 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:22.793322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:22.793337 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:22.793463 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.793468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.794154 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.794158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.794980 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:22.794989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.795641 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.795645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:22.795747 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.795775 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:41:22.795861 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:41:22.795934 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":33,"thread_start_us":104,"threads_started":1}
I20260927 00:41:22.796522 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8322ea159f4cac803d41320e1f6c17" candidate_uuid: "3fe3b0109610481083fce340eeb5d641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27430433eef4bb393af93cece29f310" is_pre_election: true
I20260927 00:41:22.796731 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8322ea159f4cac803d41320e1f6c17" candidate_uuid: "3fe3b0109610481083fce340eeb5d641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3f14c649524badaa06ea44a554ccc4" is_pre_election: true
I20260927 00:41:22.796824 1107 raft_consensus.cc:2454] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe3b0109610481083fce340eeb5d641 in term 0.
I20260927 00:41:22.797019 1375 raft_consensus.cc:2454] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fe3b0109610481083fce340eeb5d641 in term 0.
I20260927 00:41:22.797343 1175 leader_election.cc:302] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe3b0109610481083fce340eeb5d641, f27430433eef4bb393af93cece29f310; no voters:
I20260927 00:41:22.797600 1445 raft_consensus.cc:2781] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:22.797690 1445 raft_consensus.cc:482] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:22.797749 1445 raft_consensus.cc:3037] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:41:22.798703 1154 tablet.cc:2369] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:22.798727 1445 raft_consensus.cc:504] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.799099 1445 leader_election.cc:288] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [CANDIDATE]: Term 1 election: Requested vote from peers f27430433eef4bb393af93cece29f310 (127.0.98.1:34091), ae3f14c649524badaa06ea44a554ccc4 (127.0.98.3:37251)
I20260927 00:41:22.799394 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8322ea159f4cac803d41320e1f6c17" candidate_uuid: "3fe3b0109610481083fce340eeb5d641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f27430433eef4bb393af93cece29f310"
I20260927 00:41:22.799475 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc8322ea159f4cac803d41320e1f6c17" candidate_uuid: "3fe3b0109610481083fce340eeb5d641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3f14c649524badaa06ea44a554ccc4"
I20260927 00:41:22.799530 1107 raft_consensus.cc:3037] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:22.799623 1375 raft_consensus.cc:3037] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:22.800578 1107 raft_consensus.cc:2454] T dc8322ea159f4cac803d41320e1f6c17 P f27430433eef4bb393af93cece29f310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe3b0109610481083fce340eeb5d641 in term 1.
I20260927 00:41:22.800647 1375 raft_consensus.cc:2454] T dc8322ea159f4cac803d41320e1f6c17 P ae3f14c649524badaa06ea44a554ccc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fe3b0109610481083fce340eeb5d641 in term 1.
I20260927 00:41:22.800945 1175 leader_election.cc:302] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe3b0109610481083fce340eeb5d641, f27430433eef4bb393af93cece29f310; no voters:
I20260927 00:41:22.801082 1445 raft_consensus.cc:2781] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:22.801275 1445 raft_consensus.cc:686] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe3b0109610481083fce340eeb5d641, State: Running, Role: LEADER
I20260927 00:41:22.801581 1445 consensus_queue.cc:237] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } }
I20260927 00:41:22.804625 960 catalog_manager.cc:5495] T dc8322ea159f4cac803d41320e1f6c17 P 3fe3b0109610481083fce340eeb5d641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fe3b0109610481083fce340eeb5d641 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3fe3b0109610481083fce340eeb5d641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f27430433eef4bb393af93cece29f310" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae3f14c649524badaa06ea44a554ccc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe3b0109610481083fce340eeb5d641" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45085 } health_report { overall_health: HEALTHY } } }
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:22.829118 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.820509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.0.98.1:34091)
0927 00:41:22.820659 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.820663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.820828 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:41:22.820989 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.820992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.821033 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.821103 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.821734 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.822224 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.823017 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.823219 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.823320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.825767 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.825776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.825789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.825810 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.827328 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.827709 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.827715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.827716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.827805 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.828178 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.828181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.828182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.828423 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.828589 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.828775 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.828867 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:22.855158 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.846755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.98.1:34091)
0927 00:41:22.846891 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.846896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.847063 (+ 167us) server_negotiation.cc:407] Connection header received
0927 00:41:22.847168 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.847170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.847210 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.847278 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.847937 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.848416 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.849315 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.849588 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.849669 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.852365 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.852368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.852370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.852391 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.853681 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.854065 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.854067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.854068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.854136 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.854385 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.854387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.854388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.854533 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.854628 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.854872 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.854973 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:22.879789 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:22.871528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.0.98.1:34091)
0927 00:41:22.871668 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:22.871673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:22.871842 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:41:22.871969 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:22.871972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:22.872013 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:22.872082 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:22.872686 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.873237 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.874132 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:22.874356 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:22.874438 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:22.877175 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:22.877178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:22.877180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:22.877203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:22.878410 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.878789 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.878792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.878793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.878862 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:22.879086 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:22.879088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:22.879090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:22.879238 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:22.879340 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:22.879512 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:41:22.879599 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 00:41:22.881206 1067 tablet_service.cc:2908] Rejecting scan request for tablet dc8322ea159f4cac803d41320e1f6c17: Uninitialized: clean time has not yet been initialized
I20260927 00:41:22.883164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1024
I20260927 00:41:22.888700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1158
I20260927 00:41:22.893945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1292
I20260927 00:41:22.899067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 930
2026-09-27T00:41:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2195 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[1462](info): set up 2 sockets
Sep 27 00:41:22 dist-test-slave-pkws krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 00:41:23.932699 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:24.052289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:24.158722 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:24.158998 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:24.159083 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:24.163112 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:24.163192 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:41:24.163223 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:24.163265 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:24.163305 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:24.163342 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:24.168365 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:24.169413 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:24.170469 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:24.178510 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.178526 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.178517 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.178920 1475 server_base.cc:1034] running on GCE node
I20260927 00:41:24.179390 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:24.180405 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:24.181584 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790469684181567 us; error 32 us; skew 500 ppm
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:24.190132 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:24.191009 1475 webserver.cc:460] Webserver started at http://127.0.98.62:33733/ using document root <none> and password file <none>
I20260927 00:41:24.191505 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:24.191571 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:24.191723 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:24.193732 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "cde25b63815d40148e8f725e299cf7d7"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "64f5981cfa0dd69022132bc9b2ca2904"
server_key_iv: "b0da7beeafee0789b9e9458da48df937"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.194219 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "cde25b63815d40148e8f725e299cf7d7"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "64f5981cfa0dd69022132bc9b2ca2904"
server_key_iv: "b0da7beeafee0789b9e9458da48df937"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.197781 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:24.200021 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.201285 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:24.201426 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "cde25b63815d40148e8f725e299cf7d7"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "64f5981cfa0dd69022132bc9b2ca2904"
server_key_iv: "b0da7beeafee0789b9e9458da48df937"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.201535 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:24.228103 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:24.231124 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:24.231328 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:24.239300 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40529
I20260927 00:41:24.239353 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40529 every 8 connection(s)
I20260927 00:41:24.240285 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:24.243033 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:24.248247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1475
I20260927 00:41:24.248283 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Bootstrap starting.
I20260927 00:41:24.248386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:24.248651 392 external_mini_cluster.cc:1428] Setting key 4edfb236d027fcba083901e398e0032e
I20260927 00:41:24.250725 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:24.251443 1553 log.cc:824] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:24.253228 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: No bootstrap required, opened a new log
I20260927 00:41:24.256654 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 00:41:24.256865 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:24.256959 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde25b63815d40148e8f725e299cf7d7, State: Initialized, Role: FOLLOWER
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:24.257455 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 00:41:24.257606 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:24.257675 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:24.257797 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:24.258869 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 00:41:24.259196 1553 leader_election.cc:302] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde25b63815d40148e8f725e299cf7d7; no voters:
I20260927 00:41:24.259516 1553 leader_election.cc:288] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:24.259608 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:24.259822 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [term 1 LEADER]: Becoming Leader. State: Replica: cde25b63815d40148e8f725e299cf7d7, State: Running, Role: LEADER
I20260927 00:41:24.260211 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } }
I20260927 00:41:24.260733 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:24.263655 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde25b63815d40148e8f725e299cf7d7. Latest consensus state: current_term: 1 leader_uuid: "cde25b63815d40148e8f725e299cf7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } } }
I20260927 00:41:24.263458 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde25b63815d40148e8f725e299cf7d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde25b63815d40148e8f725e299cf7d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40529 } } }
I20260927 00:41:24.263464 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.249989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.98.62:40529)
0927 00:41:24.250396 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.250410 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.250443 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:41:24.251119 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.251137 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.251451 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.251813 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.252577 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.253659 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.254505 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.254957 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.255141 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.257676 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.257685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.257700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.257730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.259640 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.260205 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.260210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.260216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.260347 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.260681 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.260684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.260686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.261197 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.261375 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.262169 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.262439 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
I20260927 00:41:24.263793 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:24.263787 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7 [sys.catalog]: This master's current role is: LEADER
W20260927 00:41:24.265983 1573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:24.266117 1573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:24.266356 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:24.267024 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:24.272332 1574 catalog_manager.cc:1347] Generated new cluster ID: 9e28f7b5b5054d9ab520e4777532724a
I20260927 00:41:24.272446 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:24.288589 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:24.289834 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:24.296464 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Generated new TSK 0
I20260927 00:41:24.297178 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:24.328189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:24.437023 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:24.437253 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:24.437294 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:24.441367 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:24.441416 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:24.441485 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:24.441553 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:24.446501 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:24.447469 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:24.448449 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:24.457809 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.457810 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.457974 1579 server_base.cc:1034] running on GCE node
W20260927 00:41:24.457804 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.458407 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:24.459039 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:24.460235 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790469684460219 us; error 39 us; skew 500 ppm
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:24.463086 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:24.463938 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45501/ using document root <none> and password file <none>
I20260927 00:41:24.464473 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:24.464560 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:24.464751 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:24.466760 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f4466cdb54746c6b632d659a2f5d731"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "cd806bd2aefa74e24e4d7725653b1e5d"
server_key_iv: "c9a465a5e185cba26a211b76425da732"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.467337 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f4466cdb54746c6b632d659a2f5d731"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "cd806bd2aefa74e24e4d7725653b1e5d"
server_key_iv: "c9a465a5e185cba26a211b76425da732"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.470816 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:24.473162 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.474308 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:24.474448 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "7f4466cdb54746c6b632d659a2f5d731"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "cd806bd2aefa74e24e4d7725653b1e5d"
server_key_iv: "c9a465a5e185cba26a211b76425da732"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.474557 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:24.483821 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:24.486438 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:24.486624 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:24.487957 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:24.488032 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.488130 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:24.488159 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:24.498772 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42217
I20260927 00:41:24.499423 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42217 every 8 connection(s)
I20260927 00:41:24.499670 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:24.501371 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.488592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34017 (local address 127.0.98.62:40529)
0927 00:41:24.488729 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.488734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.489616 (+ 882us) server_negotiation.cc:407] Connection header received
0927 00:41:24.490522 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.490525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.490588 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.490672 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.491826 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.492666 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.494138 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.494361 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.494476 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.497741 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.497747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.497750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.497780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.499270 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.499839 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.499841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.499843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.499915 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.500318 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.500321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.500323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.500545 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.500646 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.501159 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.501257 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:41:24.502051 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.489031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.1:34017)
0927 00:41:24.489455 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.489501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.490322 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.490940 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.490947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.491240 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.491668 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.491680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.492806 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.492810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.493993 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.494002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.494507 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.494511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.494683 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.495230 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.495275 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.497538 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.499383 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.499388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.499400 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.499733 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.500019 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.500023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.500025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.500194 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.500637 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.500641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.500939 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.501103 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.501322 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":130,"threads_started":1}
I20260927 00:41:24.503156 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 00:41:24.503278 1710 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:24.503793 1710 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 00:41:24.504074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1579
I20260927 00:41:24.504192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:24.504516 392 external_mini_cluster.cc:1428] Setting key e7aa41f884d05ec864675d0f4f113477
I20260927 00:41:24.505367 1506 ts_manager.cc:194] Registered new tserver with Master: 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217)
I20260927 00:41:24.506660 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34017
I20260927 00:41:24.513420 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.505394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.1:34657)
0927 00:41:24.505827 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.505843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.505998 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.506261 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.506264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.506476 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.506624 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.506634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.507601 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.507605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.508409 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.508419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.508956 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.508962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.509086 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.509541 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.509559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.510002 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.511931 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.511933 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.511934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.512125 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.512425 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.512427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.512428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.512509 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.513012 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.513015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.513094 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.513189 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.513286 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":98,"threads_started":1}
I20260927 00:41:24.513692 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.505452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34657 (local address 127.0.98.62:40529)
0927 00:41:24.505577 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.505582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.505925 (+ 343us) server_negotiation.cc:407] Connection header received
0927 00:41:24.506106 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.506109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.506149 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.506230 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.506751 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.507454 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.508545 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.508812 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.508934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.510135 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.510138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.510140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.510160 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.511810 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.512224 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.512230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.512234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.512326 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.512603 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.512609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.512613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.512854 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.512994 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.513270 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.513385 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:24.532414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:24.639873 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:24.640148 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:24.640194 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:24.644243 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:24.644299 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:24.644338 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:24.644419 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:24.649549 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:24.650556 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:24.651541 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:24.660925 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.660925 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.661245 1719 server_base.cc:1034] running on GCE node
W20260927 00:41:24.661264 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.661948 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:24.662551 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:24.663759 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790469684663736 us; error 46 us; skew 500 ppm
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:24.666471 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:24.667264 1719 webserver.cc:460] Webserver started at http://127.0.98.2:35637/ using document root <none> and password file <none>
I20260927 00:41:24.667732 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:24.667786 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:24.667933 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:24.669838 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "8ec6dda065bd4844ff66faaee8eeb95a"
server_key_iv: "764099ddacdeafd2bb935d9207cdc9ef"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.670320 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "8ec6dda065bd4844ff66faaee8eeb95a"
server_key_iv: "764099ddacdeafd2bb935d9207cdc9ef"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.673781 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:24.676079 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.677315 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:24.677439 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "8ec6dda065bd4844ff66faaee8eeb95a"
server_key_iv: "764099ddacdeafd2bb935d9207cdc9ef"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.677541 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:24.687553 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:24.690375 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:24.690568 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:24.691962 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:24.692011 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.692049 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:24.692063 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:24.703186 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38323
I20260927 00:41:24.703217 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38323 every 8 connection(s)
I20260927 00:41:24.704124 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:24.706410 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.692828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56771 (local address 127.0.98.62:40529)
0927 00:41:24.692970 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.692974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.693989 (+ 1015us) server_negotiation.cc:407] Connection header received
0927 00:41:24.695072 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.695077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.695144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.695242 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.696629 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.697237 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.698556 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.698866 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.698985 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.702525 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.702528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.702531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.702559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.704196 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.704824 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.704829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.704834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.704950 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.705366 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.705369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.705371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.705555 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.705666 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.706128 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.706244 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:41:24.707089 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.693360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.2:56771)
0927 00:41:24.693839 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.693878 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.694851 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.695455 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.695461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.695952 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.696452 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.696472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.697358 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.697361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.698422 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.698431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.699021 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.699026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.699202 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.699770 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.699793 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.702368 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.704324 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.704329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.704341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.704694 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.705056 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.705061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.705063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.705245 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.705661 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.705667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.705894 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.706070 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.706361 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260927 00:41:24.707757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1719
I20260927 00:41:24.707864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:24.708113 392 external_mini_cluster.cc:1428] Setting key a4ecf78a4f97626ed54cd084c2c49370
I20260927 00:41:24.708243 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 00:41:24.708364 1850 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:24.708890 1850 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 00:41:24.709874 1507 ts_manager.cc:194] Registered new tserver with Master: d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323)
I20260927 00:41:24.710621 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56771
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:24.718412 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.710926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59095 (local address 127.0.98.62:40529)
0927 00:41:24.711386 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.711389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.711400 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:41:24.711426 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.711428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.711458 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.711551 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.711958 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.712447 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.713776 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.714068 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.714153 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.715371 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.715374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.715376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.715396 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.716775 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.717278 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.717280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.717282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.717351 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.717652 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.717654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.717655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.717834 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.717909 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.718222 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.718309 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260927 00:41:24.718510 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.710405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.2:59095)
0927 00:41:24.710689 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.710707 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.710791 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.711566 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.711569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.711756 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.711849 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.711854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.712583 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.712589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.713649 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.713662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.714200 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.714206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.714311 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.714748 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.714765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.715226 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.716916 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.716920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.716922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.717179 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.717449 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.717454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.717455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.717553 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.717928 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.717933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.718036 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.718179 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.718331 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:24.733564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40529
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:24.841140 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:24.841378 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:24.841446 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:24.845376 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:24.845427 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:24.845491 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:24.845568 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:24.850617 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:24.851627 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:24.852622 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:24.861685 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.864004 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:24.864326 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:24.864936 1859 server_base.cc:1034] running on GCE node
I20260927 00:41:24.865331 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:24.865931 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:24.867164 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790469684867135 us; error 46 us; skew 500 ppm
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:24.870004 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:24.870875 1859 webserver.cc:460] Webserver started at http://127.0.98.3:46715/ using document root <none> and password file <none>
I20260927 00:41:24.871423 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:24.871510 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:24.871731 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:24.873747 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "48b99498e16a4036964e4f9bd67209f1"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "0b81ac644e080f0238f3f21d06eded4c"
server_key_iv: "8a13df491fe6cc31e84b309685f4c3f0"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.874295 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48b99498e16a4036964e4f9bd67209f1"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "0b81ac644e080f0238f3f21d06eded4c"
server_key_iv: "8a13df491fe6cc31e84b309685f4c3f0"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.877859 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:24.880225 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.881551 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:41:24.881645 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "48b99498e16a4036964e4f9bd67209f1"
format_stamp: "Formatted at 2026-09-27 00:41:24 on dist-test-slave-pkws"
server_key: "0b81ac644e080f0238f3f21d06eded4c"
server_key_iv: "8a13df491fe6cc31e84b309685f4c3f0"
server_key_version: "encryptionkey@0"
I20260927 00:41:24.881775 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:24.913112 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:24.916424 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:24.916628 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:24.918097 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:24.918143 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:24.918181 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:24.918236 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:24.928978 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46591
I20260927 00:41:24.929054 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46591 every 8 connection(s)
I20260927 00:41:24.929891 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:24.931849 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.919015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50111 (local address 127.0.98.62:40529)
0927 00:41:24.919175 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.919179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.920037 (+ 858us) server_negotiation.cc:407] Connection header received
0927 00:41:24.921212 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.921218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.921282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.921401 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.922607 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.923161 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.924359 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.924625 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.924752 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.927875 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.927878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.927880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.927904 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.929462 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.930281 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.930285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.930287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.930399 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.930809 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.930812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.930814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.931023 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.931131 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.931626 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.931730 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:41:24.932641 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.919426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.3:50111)
0927 00:41:24.919859 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.919907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.920956 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.921607 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.921631 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.922009 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.922459 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.922471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.923284 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.923288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.924229 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.924239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.924793 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.924796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.925013 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.925595 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.925641 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.927697 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.929579 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.929584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.929598 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.930151 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.930523 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.930527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.930529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.930688 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.931149 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.931155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.931419 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.931585 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.931820 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":130,"threads_started":1}
I20260927 00:41:24.933848 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40529
I20260927 00:41:24.934003 1990 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:24.934605 1990 heartbeater.cc:511] Master 127.0.98.62:40529 requested a full tablet report, sending...
I20260927 00:41:24.935704 1506 ts_manager.cc:194] Registered new tserver with Master: 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3:46591)
I20260927 00:41:24.936479 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50111
I20260927 00:41:24.939265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 1859
I20260927 00:41:24.939411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:24.939690 392 external_mini_cluster.cc:1428] Setting key 21ab864e6422252812d9d8372cc7c766
I20260927 00:41:24.941733 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:24.942787 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.935813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.98.3:53751)
0927 00:41:24.936048 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.936061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.936136 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.936365 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.936367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.936514 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:24.936597 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.936601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.937535 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.937538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.938098 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.938105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.938604 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.938606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.938680 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.939162 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:24.939173 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:24.939573 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:24.941341 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.941344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.941346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.941597 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.941873 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:24.941875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:24.941877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:24.941961 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:24.942376 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:24.942378 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:24.942448 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.942534 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.942653 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260927 00:41:24.943089 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.935858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53751 (local address 127.0.98.62:40529)
0927 00:41:24.935959 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.935963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.936130 (+ 167us) server_negotiation.cc:407] Connection header received
0927 00:41:24.936221 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.936224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.936272 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.936344 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.936690 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.937415 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.938216 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.938506 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.938591 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.939700 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.939707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.939711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.939737 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.941237 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.941701 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.941705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.941707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.941793 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.942046 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.942049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.942051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.942277 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.942379 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.942619 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.942743 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:41:24 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:24.975993 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.966460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.0.98.62:40529)
0927 00:41:24.966626 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.966630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.966651 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:24.966715 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.966717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.966785 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.966895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:24.967884 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.968627 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.970056 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.970368 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.970450 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.972871 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:24.972874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:24.972876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:24.972953 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:24.974423 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.974817 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.974820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.974821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.974889 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:24.975164 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:24.975166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:24.975167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:24.975321 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:24.975420 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.975676 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.975827 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:41:24.979009 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:24.982190 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:24.996248 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.990236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46591 (local address 127.0.0.1:60104)
0927 00:41:24.990997 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.991008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.991074 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.992373 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.992376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.992394 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:24.992468 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.992474 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993726 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.993729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.994670 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.994679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.995704 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.995707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.995786 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.995859 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.995974 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.996092 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260927 00:41:24.996254 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.990710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.0.98.3:46591)
0927 00:41:24.991892 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.991926 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.991947 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:24.992017 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.992024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.992236 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.992386 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:24.992583 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993584 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.994793 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.995555 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.995672 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.995840 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.996018 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.996111 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":108,"threads_started":1}
I20260927 00:41:24.996287 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.989952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38323 (local address 127.0.0.1:46024)
0927 00:41:24.990620 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.990669 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.990792 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.991685 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.991691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.991715 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:24.991838 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.991844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993325 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.993329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.994863 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.994875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.995881 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.995884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.995984 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.996004 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.996124 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.996183 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260927 00:41:24.997053 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.990145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.2:38323)
0927 00:41:24.991204 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.991217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.991247 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:41:24.991345 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.991353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.991535 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.991665 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:24.991974 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993206 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.995002 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.995754 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.995866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.996026 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.996445 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.996560 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":124,"threads_started":1}
I20260927 00:41:24.997677 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.990315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42217 (local address 127.0.0.1:58206)
0927 00:41:24.991253 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:24.991264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:24.991342 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:24.992009 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:24.992012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:24.992027 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:24.992125 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.992133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993370 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.993373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.994863 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:24.994875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.997261 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.997266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:24.997369 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.997402 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:41:24.997470 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 00:41:24.997543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":100,"thread_start_us":68,"threads_started":1}
I20260927 00:41:24.998139 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:24.990728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.98.1:42217)
0927 00:41:24.991586 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:24.991599 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:24.991642 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:41:24.991711 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:24.991716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:24.991895 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:24.992037 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:24.992248 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.993232 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.996083 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:24.996879 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:24.997028 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:24.997197 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:24.997953 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 00:41:24.998026 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":77,"threads_started":1}
I20260927 00:41:24.999583 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 367cb822fcb94b479da4ce321a8b34cb (DEFAULT_TABLE table=test-table [id=24b54fb27cca4a408dce77cd88735cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.000743 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367cb822fcb94b479da4ce321a8b34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.000720 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 367cb822fcb94b479da4ce321a8b34cb (DEFAULT_TABLE table=test-table [id=24b54fb27cca4a408dce77cd88735cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.001751 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367cb822fcb94b479da4ce321a8b34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.001988 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 367cb822fcb94b479da4ce321a8b34cb (DEFAULT_TABLE table=test-table [id=24b54fb27cca4a408dce77cd88735cba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.002815 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367cb822fcb94b479da4ce321a8b34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.008689 2014 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Bootstrap starting.
I20260927 00:41:25.010104 2013 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Bootstrap starting.
I20260927 00:41:25.011247 2014 tablet_bootstrap.cc:654] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.012099 2014 log.cc:824] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:25.012566 2013 tablet_bootstrap.cc:654] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.013223 2015 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Bootstrap starting.
I20260927 00:41:25.013411 2013 log.cc:824] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:25.014091 2014 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: No bootstrap required, opened a new log
I20260927 00:41:25.014302 2014 ts_tablet_manager.cc:1397] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:25.015352 2013 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: No bootstrap required, opened a new log
I20260927 00:41:25.015352 2015 tablet_bootstrap.cc:654] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.015537 2013 ts_tablet_manager.cc:1397] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:25.016134 2015 log.cc:824] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:25.017656 2014 raft_consensus.cc:348] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.017882 2014 raft_consensus.cc:374] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.017935 2014 raft_consensus.cc:729] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.018054 2015 tablet_bootstrap.cc:492] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: No bootstrap required, opened a new log
I20260927 00:41:25.018388 2015 ts_tablet_manager.cc:1397] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:41:25.018457 2014 consensus_queue.cc:260] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.019306 2014 ts_tablet_manager.cc:1428] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 00:41:25.019223 2013 raft_consensus.cc:348] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.019457 2013 raft_consensus.cc:374] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.019513 2013 raft_consensus.cc:729] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4466cdb54746c6b632d659a2f5d731, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.019661 1990 heartbeater.cc:503] Master 127.0.98.62:40529 was elected leader, sending a full tablet report...
I20260927 00:41:25.019943 2013 consensus_queue.cc:260] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.020516 1710 heartbeater.cc:503] Master 127.0.98.62:40529 was elected leader, sending a full tablet report...
I20260927 00:41:25.020913 2013 ts_tablet_manager.cc:1428] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 00:41:25.022426 2015 raft_consensus.cc:348] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.022694 2015 raft_consensus.cc:374] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.022763 2015 raft_consensus.cc:729] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3caa6ed79f54d8ca6be91dfcafffa76, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.023149 2015 consensus_queue.cc:260] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.023763 1850 heartbeater.cc:503] Master 127.0.98.62:40529 was elected leader, sending a full tablet report...
I20260927 00:41:25.024030 2015 ts_tablet_manager.cc:1428] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:41:25.068636 2019 raft_consensus.cc:482] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:25.068868 2019 raft_consensus.cc:504] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.070063 2019 leader_election.cc:288] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323), 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217)
I20260927 00:41:25.074331 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.070397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35849 (local address 127.0.98.1:42217)
0927 00:41:25.070567 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.070572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.070616 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:41:25.070766 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.070772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.070831 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.070907 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:25.071228 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.072022 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.073249 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.073804 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.073885 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.073937 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.074148 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.074203 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:41:25.074368 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.070313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42217 (local address 127.0.98.3:35849)
0927 00:41:25.070539 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:25.070555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:25.070638 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:25.070932 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:25.070935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:25.070953 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:25.071099 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.071106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.072126 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.072145 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.073118 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.073126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.073917 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.073920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.074025 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.074049 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:41:25.074127 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:41:25.074212 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 00:41:25.074895 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.070406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41227 (local address 127.0.98.2:38323)
0927 00:41:25.070672 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.070678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.070807 (+ 129us) server_negotiation.cc:407] Connection header received
0927 00:41:25.070942 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.070945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.070994 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.071075 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:25.071807 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.072430 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.073682 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.074218 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.074313 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.074368 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.074716 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.074787 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260927 00:41:25.074900 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.070313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38323 (local address 127.0.98.3:41227)
0927 00:41:25.070709 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:25.070725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:25.070832 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:25.071112 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:25.071115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:25.071131 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:25.071690 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.071699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.072544 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.072546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.073553 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.073563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.074331 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.074334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.074592 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.074610 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:41:25.074675 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:41:25.074719 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":32,"thread_start_us":116,"threads_started":1}
I20260927 00:41:25.075155 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367cb822fcb94b479da4ce321a8b34cb" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731" is_pre_election: true
I20260927 00:41:25.075479 1662 raft_consensus.cc:2454] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.075510 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367cb822fcb94b479da4ce321a8b34cb" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" is_pre_election: true
I20260927 00:41:25.075775 1804 raft_consensus.cc:2454] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.075979 1875 leader_election.cc:302] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, 7f4466cdb54746c6b632d659a2f5d731; no voters:
I20260927 00:41:25.076258 2019 raft_consensus.cc:2781] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:25.076336 2019 raft_consensus.cc:482] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:25.076380 2019 raft_consensus.cc:3037] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.077471 2019 raft_consensus.cc:504] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.077769 2019 leader_election.cc:288] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Requested vote from peers d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323), 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217)
I20260927 00:41:25.078069 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367cb822fcb94b479da4ce321a8b34cb" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
I20260927 00:41:25.078109 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367cb822fcb94b479da4ce321a8b34cb" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731"
I20260927 00:41:25.078197 1804 raft_consensus.cc:3037] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.078238 1662 raft_consensus.cc:3037] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.079273 1804 raft_consensus.cc:2454] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.079304 1662 raft_consensus.cc:2454] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.079578 1878 leader_election.cc:302] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, d3caa6ed79f54d8ca6be91dfcafffa76; no voters:
I20260927 00:41:25.079752 2019 raft_consensus.cc:2781] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:25.079964 2019 raft_consensus.cc:686] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 1 LEADER]: Becoming Leader. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Running, Role: LEADER
I20260927 00:41:25.080287 2019 consensus_queue.cc:237] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } }
I20260927 00:41:25.083338 1507 catalog_manager.cc:5495] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:25.121332 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.117623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.0.98.3:46591)
0927 00:41:25.117772 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.117776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.117791 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:25.117978 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.117982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.118044 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.118149 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:25.118412 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.118932 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.119818 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.120058 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.120173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.120488 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:25.120874 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:25.121018 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.121137 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.121189 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:41:25.126751 1662 raft_consensus.cc:1261] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.126750 1804 raft_consensus.cc:1261] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.127449 2023 consensus_queue.cc:1035] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.127661 2019 consensus_queue.cc:1035] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.135851 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333763830272348160 to 7333763830091653120. Current Snapshot: MvccSnapshot[applied={T|T < 7333763830272348160}]
I20260927 00:41:25.136204 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333763830272348160 to 7333763830091653120. Current Snapshot: MvccSnapshot[applied={T|T < 7333763830272348160}]
I20260927 00:41:25.146373 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:25.146569 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:25.149003 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Sending DeleteTablet for 3 replicas of tablet 367cb822fcb94b479da4ce321a8b34cb
I20260927 00:41:25.149770 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 367cb822fcb94b479da4ce321a8b34cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46024
I20260927 00:41:25.149765 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 367cb822fcb94b479da4ce321a8b34cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60104
I20260927 00:41:25.150243 2036 tablet_replica.cc:330] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: stopping tablet replica
I20260927 00:41:25.150252 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 367cb822fcb94b479da4ce321a8b34cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58206
I20260927 00:41:25.150488 2037 tablet_replica.cc:330] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: stopping tablet replica
I20260927 00:41:25.150599 2036 raft_consensus.cc:2229] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:25.150826 2037 raft_consensus.cc:2229] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:25.150928 2036 raft_consensus.cc:2258] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.151305 2037 raft_consensus.cc:2258] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.151465 2038 tablet_replica.cc:330] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: stopping tablet replica
I20260927 00:41:25.151439 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:41:25.151750 2038 raft_consensus.cc:2229] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 00:41:25.151863 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:25.152035 2038 raft_consensus.cc:2258] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.153172 2037 ts_tablet_manager.cc:1905] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.153409 2036 ts_tablet_manager.cc:1905] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.156013 2036 ts_tablet_manager.cc:1918] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:25.156121 2036 log.cc:1196] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/367cb822fcb94b479da4ce321a8b34cb
I20260927 00:41:25.156255 2037 ts_tablet_manager.cc:1918] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:25.156409 2037 log.cc:1196] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/367cb822fcb94b479da4ce321a8b34cb
I20260927 00:41:25.156489 2036 ts_tablet_manager.cc:1939] T 367cb822fcb94b479da4ce321a8b34cb P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting consensus metadata
I20260927 00:41:25.156913 2037 ts_tablet_manager.cc:1939] T 367cb822fcb94b479da4ce321a8b34cb P 48b99498e16a4036964e4f9bd67209f1: Deleting consensus metadata
I20260927 00:41:25.157891 1494 catalog_manager.cc:4841] TS d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323): tablet 367cb822fcb94b479da4ce321a8b34cb (table test-table [id=24b54fb27cca4a408dce77cd88735cba]) successfully deleted
I20260927 00:41:25.157927 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (DEFAULT_TABLE table=test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.158181 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.158357 1493 catalog_manager.cc:4841] TS 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3:46591): tablet 367cb822fcb94b479da4ce321a8b34cb (table test-table [id=24b54fb27cca4a408dce77cd88735cba]) successfully deleted
I20260927 00:41:25.158581 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (DEFAULT_TABLE table=test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.158795 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.160215 2014 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Bootstrap starting.
I20260927 00:41:25.160468 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (DEFAULT_TABLE table=test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:25.160686 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.160873 2013 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Bootstrap starting.
I20260927 00:41:25.161207 2014 tablet_bootstrap.cc:654] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.161891 2013 tablet_bootstrap.cc:654] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.162361 2014 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: No bootstrap required, opened a new log
I20260927 00:41:25.162434 2014 ts_tablet_manager.cc:1397] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:25.162842 2015 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Bootstrap starting.
I20260927 00:41:25.162916 2014 raft_consensus.cc:348] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.163017 2014 raft_consensus.cc:374] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.163053 2014 raft_consensus.cc:729] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.163172 2014 consensus_queue.cc:260] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.163450 2014 ts_tablet_manager.cc:1428] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:25.163794 2015 tablet_bootstrap.cc:654] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.164541 2038 ts_tablet_manager.cc:1905] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.165364 2013 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: No bootstrap required, opened a new log
I20260927 00:41:25.165458 2013 ts_tablet_manager.cc:1397] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 00:41:25.165921 2013 raft_consensus.cc:348] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.166067 2013 raft_consensus.cc:374] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.166108 2013 raft_consensus.cc:729] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4466cdb54746c6b632d659a2f5d731, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.166240 2013 consensus_queue.cc:260] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.166520 2013 ts_tablet_manager.cc:1428] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:25.167282 2015 tablet_bootstrap.cc:492] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: No bootstrap required, opened a new log
I20260927 00:41:25.167368 2015 ts_tablet_manager.cc:1397] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:41:25.167802 2015 raft_consensus.cc:348] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.167917 2015 raft_consensus.cc:374] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.167946 2015 raft_consensus.cc:729] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3caa6ed79f54d8ca6be91dfcafffa76, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.167992 2038 ts_tablet_manager.cc:1918] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:25.168097 2038 log.cc:1196] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/367cb822fcb94b479da4ce321a8b34cb
I20260927 00:41:25.168087 2015 consensus_queue.cc:260] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.168349 2015 ts_tablet_manager.cc:1428] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:25.168395 2038 ts_tablet_manager.cc:1939] T 367cb822fcb94b479da4ce321a8b34cb P 7f4466cdb54746c6b632d659a2f5d731: Deleting consensus metadata
I20260927 00:41:25.169823 1491 catalog_manager.cc:4841] TS 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217): tablet 367cb822fcb94b479da4ce321a8b34cb (table test-table [id=24b54fb27cca4a408dce77cd88735cba]) successfully deleted
W20260927 00:41:25.181005 1991 tablet.cc:2369] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:25.192080 2032 raft_consensus.cc:482] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:25.192170 2032 raft_consensus.cc:504] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.192589 2032 leader_election.cc:288] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217), d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323)
I20260927 00:41:25.192952 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd31981bdb94b7a9d3a248a0e76fa2d" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731" is_pre_election: true
I20260927 00:41:25.192952 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd31981bdb94b7a9d3a248a0e76fa2d" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" is_pre_election: true
I20260927 00:41:25.193089 1804 raft_consensus.cc:2454] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.193095 1662 raft_consensus.cc:2454] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.193351 1878 leader_election.cc:302] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, d3caa6ed79f54d8ca6be91dfcafffa76; no voters:
I20260927 00:41:25.193506 2032 raft_consensus.cc:2781] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:25.193569 2032 raft_consensus.cc:482] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:25.193611 2032 raft_consensus.cc:3037] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.194547 2032 raft_consensus.cc:504] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.194882 2032 leader_election.cc:288] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Requested vote from peers 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217), d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323)
I20260927 00:41:25.195169 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd31981bdb94b7a9d3a248a0e76fa2d" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731"
I20260927 00:41:25.195276 1662 raft_consensus.cc:3037] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.196105 1662 raft_consensus.cc:2454] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.196337 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd31981bdb94b7a9d3a248a0e76fa2d" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
I20260927 00:41:25.196360 1875 leader_election.cc:302] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, 7f4466cdb54746c6b632d659a2f5d731; no voters:
I20260927 00:41:25.196450 1804 raft_consensus.cc:3037] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.197347 1804 raft_consensus.cc:2454] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.197659 2032 raft_consensus.cc:2781] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:25.197772 2032 raft_consensus.cc:686] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 1 LEADER]: Becoming Leader. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Running, Role: LEADER
I20260927 00:41:25.197927 2032 consensus_queue.cc:237] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.199587 1506 catalog_manager.cc:5495] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } health_report { overall_health: HEALTHY } } }
W20260927 00:41:25.205202 1851 tablet.cc:2369] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:41:25 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:25.244120 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.234536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40529 (local address 127.0.0.1:44504)
0927 00:41:25.234850 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:25.234866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:25.234974 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:25.235257 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:25.235260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:25.235772 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:25.235883 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.235890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.236638 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.236642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.237549 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.237560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.237995 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.237999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.238090 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.238662 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:25.238677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:25.240423 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:25.242471 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:25.242478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:25.242484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:25.242761 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:25.243067 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:25.243070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:25.243072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:25.243157 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:25.243574 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:25.243580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:25.243694 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:41:25.243810 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 00:41:25.243936 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260927 00:41:25.244151 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.234602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.98.62:40529)
0927 00:41:25.234733 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.234737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.234941 (+ 204us) server_negotiation.cc:407] Connection header received
0927 00:41:25.235106 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.235108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.235145 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.235206 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:25.235992 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.236519 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.237680 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.237885 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.237965 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.240562 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:25.240566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:25.240569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:25.240622 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:25.242360 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.242867 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.242871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.242873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.242959 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.243253 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.243256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.243258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.243467 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:25.243569 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.243887 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.244002 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:41:25.248243 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44504:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:41:25.249039 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260927 00:41:25.251173 1711 tablet.cc:2369] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:25.254328 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 3e158dde862c4d95bd69d5e92a951aa4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f885590232bc482a94db0178c970606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:25.254360 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 3e158dde862c4d95bd69d5e92a951aa4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f885590232bc482a94db0178c970606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:25.254680 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e158dde862c4d95bd69d5e92a951aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.254745 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e158dde862c4d95bd69d5e92a951aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.255170 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 3e158dde862c4d95bd69d5e92a951aa4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f885590232bc482a94db0178c970606c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:25.255467 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e158dde862c4d95bd69d5e92a951aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:25.257583 2013 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: Bootstrap starting.
I20260927 00:41:25.257750 2015 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: Bootstrap starting.
I20260927 00:41:25.258711 2014 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: Bootstrap starting.
I20260927 00:41:25.258965 2013 tablet_bootstrap.cc:654] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.258999 2015 tablet_bootstrap.cc:654] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.259702 2014 tablet_bootstrap.cc:654] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:25.260571 2013 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: No bootstrap required, opened a new log
I20260927 00:41:25.260658 2013 ts_tablet_manager.cc:1397] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:25.261166 2013 raft_consensus.cc:348] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.261242 2015 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: No bootstrap required, opened a new log
I20260927 00:41:25.261283 2013 raft_consensus.cc:374] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.261315 2015 ts_tablet_manager.cc:1397] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:41:25.261317 2013 raft_consensus.cc:729] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f4466cdb54746c6b632d659a2f5d731, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.261453 2014 tablet_bootstrap.cc:492] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: No bootstrap required, opened a new log
I20260927 00:41:25.261436 2013 consensus_queue.cc:260] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.261519 2014 ts_tablet_manager.cc:1397] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:25.261704 2013 ts_tablet_manager.cc:1428] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:25.261708 2015 raft_consensus.cc:348] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.261813 2015 raft_consensus.cc:374] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.261842 2015 raft_consensus.cc:729] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3caa6ed79f54d8ca6be91dfcafffa76, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.261924 2020 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.261968 2015 consensus_queue.cc:260] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.261991 2014 raft_consensus.cc:348] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.262048 2020 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.262086 2014 raft_consensus.cc:374] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:25.262118 2014 raft_consensus.cc:729] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Initialized, Role: FOLLOWER
I20260927 00:41:25.262249 2015 ts_tablet_manager.cc:1428] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:25.262228 2014 consensus_queue.cc:260] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.262454 2014 ts_tablet_manager.cc:1428] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:25.262495 2021 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.262637 2021 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.262768 2019 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.262879 2019 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.362089 2019 raft_consensus.cc:482] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:25.362229 2019 raft_consensus.cc:504] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.362591 2019 leader_election.cc:288] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217), d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323)
I20260927 00:41:25.363091 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e158dde862c4d95bd69d5e92a951aa4" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" is_pre_election: true
I20260927 00:41:25.363099 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e158dde862c4d95bd69d5e92a951aa4" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731" is_pre_election: true
I20260927 00:41:25.363252 1804 raft_consensus.cc:2454] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.363273 1662 raft_consensus.cc:2454] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 0.
I20260927 00:41:25.363600 1878 leader_election.cc:302] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, d3caa6ed79f54d8ca6be91dfcafffa76; no voters:
I20260927 00:41:25.363796 2019 raft_consensus.cc:2781] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:25.363858 2019 raft_consensus.cc:482] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:25.363899 2019 raft_consensus.cc:3037] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.364821 2019 raft_consensus.cc:504] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.365193 2019 leader_election.cc:288] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Requested vote from peers 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217), d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323)
I20260927 00:41:25.365536 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e158dde862c4d95bd69d5e92a951aa4" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f4466cdb54746c6b632d659a2f5d731"
I20260927 00:41:25.365536 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e158dde862c4d95bd69d5e92a951aa4" candidate_uuid: "48b99498e16a4036964e4f9bd67209f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76"
I20260927 00:41:25.365664 1662 raft_consensus.cc:3037] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.365664 1804 raft_consensus.cc:3037] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:25.366742 1662 raft_consensus.cc:2454] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.366742 1804 raft_consensus.cc:2454] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b99498e16a4036964e4f9bd67209f1 in term 1.
I20260927 00:41:25.367034 1875 leader_election.cc:302] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b99498e16a4036964e4f9bd67209f1, 7f4466cdb54746c6b632d659a2f5d731; no voters:
I20260927 00:41:25.367208 2019 raft_consensus.cc:2781] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:25.367321 2019 raft_consensus.cc:686] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [term 1 LEADER]: Becoming Leader. State: Replica: 48b99498e16a4036964e4f9bd67209f1, State: Running, Role: LEADER
I20260927 00:41:25.367457 2019 consensus_queue.cc:237] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } }
I20260927 00:41:25.368134 2032 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: TxnStatusTablet state changed. Reason: New leader 48b99498e16a4036964e4f9bd67209f1. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.368249 2032 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:25.368510 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:41:25.368675 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:41:25.369243 1507 catalog_manager.cc:5495] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:25.388340 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.384837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.3:46591)
0927 00:41:25.384965 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.384969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.385092 (+ 123us) server_negotiation.cc:407] Connection header received
0927 00:41:25.385281 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.385284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.385325 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.385384 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:25.385663 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.386238 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.387396 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.387592 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.387670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.387967 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:25.388028 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:25.388075 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.388203 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.388241 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:41:25.388404 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.384742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46591 (local address 127.0.0.1:60124)
0927 00:41:25.385017 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:25.385033 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:25.385152 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:25.385423 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:25.385426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:25.385434 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:25.385560 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.385566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.386321 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.386324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.387288 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.387299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.387688 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.387692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.387792 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.387842 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:25.388107 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:25.388119 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 00:41:25.388188 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:41:25.388270 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 00:41:25.390908 1662 raft_consensus.cc:1261] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.390925 1804 raft_consensus.cc:1261] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.391340 2032 consensus_queue.cc:1035] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.391474 2019 consensus_queue.cc:1035] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.392748 2021 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: TxnStatusTablet state changed. Reason: New leader 48b99498e16a4036964e4f9bd67209f1. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.392784 2020 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: TxnStatusTablet state changed. Reason: New leader 48b99498e16a4036964e4f9bd67209f1. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.392872 2021 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.392910 2020 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.394215 2023 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.394344 2023 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:25.394441 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333763831357227008 to 7333763831266193408. Current Snapshot: MvccSnapshot[applied={T|T < 7333763831357227008}]
I20260927 00:41:25.395927 2023 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.396034 2023 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 48b99498e16a4036964e4f9bd67209f1: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:25.396785 2020 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.396797 2021 tablet_replica.cc:439] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48b99498e16a4036964e4f9bd67209f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 } } peers { permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 } } peers { permanent_uuid: "48b99498e16a4036964e4f9bd67209f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46591 } } }
I20260927 00:41:25.396905 2020 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P 7f4466cdb54746c6b632d659a2f5d731: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.396921 2021 tablet_replica.cc:442] T 3e158dde862c4d95bd69d5e92a951aa4 P d3caa6ed79f54d8ca6be91dfcafffa76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:25.409997 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.406195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52463 (local address 127.0.98.3:46591)
0927 00:41:25.406314 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.406319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.406376 (+ 57us) server_negotiation.cc:407] Connection header received
0927 00:41:25.406493 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.406496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.406541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.406621 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:25.406806 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.407576 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.408511 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.408878 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.409022 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.409444 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:25.409522 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:25.409596 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.409748 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.409837 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":43}
I20260927 00:41:25.410033 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.406096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46591 (local address 127.0.98.3:52463)
0927 00:41:25.406324 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:25.406337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:25.406411 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:25.406634 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:25.406636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:25.406641 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:25.406721 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.406725 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.407691 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.407694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.408407 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:25.408416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.409031 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.409035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:25.409210 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.409272 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:25.409642 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:25.409651 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:41:25.409726 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:41:25.409838 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3}
I20260927 00:41:25.416882 1662 raft_consensus.cc:1261] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.416932 1804 raft_consensus.cc:1261] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Refusing update from remote peer 48b99498e16a4036964e4f9bd67209f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:25.417273 2023 consensus_queue.cc:1035] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3caa6ed79f54d8ca6be91dfcafffa76" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.417411 2019 consensus_queue.cc:1035] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f4466cdb54746c6b632d659a2f5d731" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:25.456115 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:25.456278 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:25.458266 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde25b63815d40148e8f725e299cf7d7: Sending DeleteTablet for 3 replicas of tablet 7fd31981bdb94b7a9d3a248a0e76fa2d
I20260927 00:41:25.458802 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58206
I20260927 00:41:25.458987 2038 tablet_replica.cc:330] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: stopping tablet replica
I20260927 00:41:25.459123 2038 raft_consensus.cc:2229] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:25.459188 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46024
I20260927 00:41:25.459233 2038 raft_consensus.cc:2258] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.459268 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd31981bdb94b7a9d3a248a0e76fa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60104
I20260927 00:41:25.459419 2036 tablet_replica.cc:330] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: stopping tablet replica
I20260927 00:41:25.459430 2037 tablet_replica.cc:330] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: stopping tablet replica
I20260927 00:41:25.459511 2036 raft_consensus.cc:2229] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:25.459532 2037 raft_consensus.cc:2229] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:25.459592 2036 raft_consensus.cc:2258] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.459694 2037 raft_consensus.cc:2258] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:25.460319 2037 ts_tablet_manager.cc:1905] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.460413 2036 ts_tablet_manager.cc:1905] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.462925 2036 ts_tablet_manager.cc:1918] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:25.463027 2036 log.cc:1196] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/7fd31981bdb94b7a9d3a248a0e76fa2d
I20260927 00:41:25.463081 2037 ts_tablet_manager.cc:1918] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:25.463165 2037 log.cc:1196] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/7fd31981bdb94b7a9d3a248a0e76fa2d
I20260927 00:41:25.463295 2036 ts_tablet_manager.cc:1939] T 7fd31981bdb94b7a9d3a248a0e76fa2d P d3caa6ed79f54d8ca6be91dfcafffa76: Deleting consensus metadata
I20260927 00:41:25.463436 2037 ts_tablet_manager.cc:1939] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 48b99498e16a4036964e4f9bd67209f1: Deleting consensus metadata
I20260927 00:41:25.463981 1494 catalog_manager.cc:4841] TS d3caa6ed79f54d8ca6be91dfcafffa76 (127.0.98.2:38323): tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (table test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]) successfully deleted
I20260927 00:41:25.464033 1493 catalog_manager.cc:4841] TS 48b99498e16a4036964e4f9bd67209f1 (127.0.98.3:46591): tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (table test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]) successfully deleted
I20260927 00:41:25.465368 2038 ts_tablet_manager.cc:1905] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:25.467905 2038 ts_tablet_manager.cc:1918] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:25.467986 2038 log.cc:1196] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/7fd31981bdb94b7a9d3a248a0e76fa2d
I20260927 00:41:25.468246 2038 ts_tablet_manager.cc:1939] T 7fd31981bdb94b7a9d3a248a0e76fa2d P 7f4466cdb54746c6b632d659a2f5d731: Deleting consensus metadata
I20260927 00:41:25.469578 1491 catalog_manager.cc:4841] TS 7f4466cdb54746c6b632d659a2f5d731 (127.0.98.1:42217): tablet 7fd31981bdb94b7a9d3a248a0e76fa2d (table test-table [id=3d0da59d7a6b45df86956d5f8e106d1d]) successfully deleted
Sep 27 00:41:25 dist-test-slave-pkws krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:25.475445 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.465059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.0.98.1:42217)
0927 00:41:25.465180 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.465185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.465592 (+ 407us) server_negotiation.cc:407] Connection header received
0927 00:41:25.465750 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.465753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.465803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.465886 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:25.466559 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.467066 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.467884 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.468239 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.468336 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.471933 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:25.471943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:25.471959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:25.471990 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:25.473712 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.474154 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.474161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.474163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.474245 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.474527 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.474532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.474533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.474774 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:25.474988 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.475170 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.475289 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:41:25.475857 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58210
I20260927 00:41:25.485890 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:25.477419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.98.62:40529)
0927 00:41:25.477552 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:25.477556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:25.477702 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:41:25.477909 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:25.477911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:25.477950 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:25.478013 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:25.478807 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.479368 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.480494 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:25.480933 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:25.481055 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:25.482505 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:25.482509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:25.482512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:25.482538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:25.484060 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.484570 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.484575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.484577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.484663 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:25.485004 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:25.485007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:25.485008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:25.485209 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:25.485349 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:25.485638 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 00:41:25.485746 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:41:25.486271 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44518
I20260927 00:41:25.487416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1579
I20260927 00:41:25.494189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1719
I20260927 00:41:25.500768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1859
I20260927 00:41:25.506170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 1475
2026-09-27T00:41:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2606 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:25 dist-test-slave-pkws krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:25 dist-test-slave-pkws krb5kdc[2070](info): set up 2 sockets
Sep 27 00:41:25 dist-test-slave-pkws krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260927 00:41:26.548632 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:26 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:26.667419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:26.773512 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:26.773823 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:26.773903 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:26.778000 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:26.778080 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:41:26.778113 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:26.778157 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:26.778194 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:26.778234 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:26.784049 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35503
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:26.785158 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:26.786170 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:26.794445 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:26.794445 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:26.794477 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:26.794770 2083 server_base.cc:1034] running on GCE node
I20260927 00:41:26.795547 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:26.796602 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:26.797806 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790469686797779 us; error 48 us; skew 500 ppm
Sep 27 00:41:26 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:26.800482 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:26.801378 2083 webserver.cc:460] Webserver started at http://127.0.98.62:35867/ using document root <none> and password file <none>
I20260927 00:41:26.801867 2083 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:26.801932 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:26.802161 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:26.804072 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "26d3786261fa4fadb2d8ca38a6323ab5"
format_stamp: "Formatted at 2026-09-27 00:41:26 on dist-test-slave-pkws"
server_key: "9a79868010b23413b8bebb7152081176"
server_key_iv: "11071cd93174771fcd834175d9426f30"
server_key_version: "encryptionkey@0"
I20260927 00:41:26.804682 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "26d3786261fa4fadb2d8ca38a6323ab5"
format_stamp: "Formatted at 2026-09-27 00:41:26 on dist-test-slave-pkws"
server_key: "9a79868010b23413b8bebb7152081176"
server_key_iv: "11071cd93174771fcd834175d9426f30"
server_key_version: "encryptionkey@0"
I20260927 00:41:26.808281 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:26.810628 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:26.811789 2083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:41:26.811911 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "26d3786261fa4fadb2d8ca38a6323ab5"
format_stamp: "Formatted at 2026-09-27 00:41:26 on dist-test-slave-pkws"
server_key: "9a79868010b23413b8bebb7152081176"
server_key_iv: "11071cd93174771fcd834175d9426f30"
server_key_version: "encryptionkey@0"
I20260927 00:41:26.812027 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:26.823720 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:26.827415 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:26.827622 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:26.836973 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35503
I20260927 00:41:26.836987 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35503 every 8 connection(s)
I20260927 00:41:26.837854 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:26.840492 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:26.842643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2083
I20260927 00:41:26.842741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:26.843000 392 external_mini_cluster.cc:1428] Setting key b053acaa3a981e399294915b78223b5c
I20260927 00:41:26.846397 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Bootstrap starting.
I20260927 00:41:26.848534 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:26.849316 2161 log.cc:824] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:26.851131 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: No bootstrap required, opened a new log
Sep 27 00:41:26 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:26.854722 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } }
I20260927 00:41:26.854944 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:26.855014 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d3786261fa4fadb2d8ca38a6323ab5, State: Initialized, Role: FOLLOWER
I20260927 00:41:26.855633 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } }
I20260927 00:41:26.855804 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:26.855930 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:26.856055 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:26.856848 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:26.844342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.62:35503)
0927 00:41:26.844769 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:26.844783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:26.844816 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:41:26.845525 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:26.845541 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:26.845865 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:26.846203 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:26.846990 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:26.847967 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:26.848885 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:26.849383 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:26.849559 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:26.852176 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:26.852181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:26.852194 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:26.852218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:26.853965 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:26.854403 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:26.854413 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:26.854418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:26.854523 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:26.854812 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:26.854816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:26.854817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:26.855291 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:26.855493 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:26.855798 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:41:26.856113 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":137,"threads_started":1}
I20260927 00:41:26.856932 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } }
I20260927 00:41:26.857286 2161 leader_election.cc:302] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d3786261fa4fadb2d8ca38a6323ab5; no voters:
I20260927 00:41:26.857597 2161 leader_election.cc:288] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:26.857697 2166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:26.857949 2166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 26d3786261fa4fadb2d8ca38a6323ab5, State: Running, Role: LEADER
I20260927 00:41:26.858309 2166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } }
I20260927 00:41:26.858749 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:26.859917 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } } }
I20260927 00:41:26.860042 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:26.860392 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:26.860373 2167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d3786261fa4fadb2d8ca38a6323ab5. Latest consensus state: current_term: 1 leader_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d3786261fa4fadb2d8ca38a6323ab5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35503 } } }
I20260927 00:41:26.860764 2167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:26.863183 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:26.868417 2175 catalog_manager.cc:1347] Generated new cluster ID: 10439fad189a41c19ba5bfdf28027fa8
I20260927 00:41:26.868487 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:26.877882 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:26.878737 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:26.885957 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Generated new TSK 0
I20260927 00:41:26.886600 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:26.912063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35503
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:27.020409 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:27.020633 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:27.020677 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:27.024683 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:27.024735 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:27.024797 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:41:27.024828 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:27.024879 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:27.030022 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:27.030929 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:27.031869 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:27.040947 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.041028 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.040931 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.041177 2187 server_base.cc:1034] running on GCE node
I20260927 00:41:27.041538 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:27.042131 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:27.043324 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790469687043316 us; error 25 us; skew 500 ppm
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:27.046173 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:27.046967 2187 webserver.cc:460] Webserver started at http://127.0.98.1:39741/ using document root <none> and password file <none>
I20260927 00:41:27.047501 2187 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:27.047591 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:27.047808 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:27.049791 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cb17aa9843645e1abd0e5ff90dd07bb"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "581f5c398dfc16cb32770ac7e6c9e799"
server_key_iv: "d5df6a3009880d860f75e637ba8c76fc"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.050333 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cb17aa9843645e1abd0e5ff90dd07bb"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "581f5c398dfc16cb32770ac7e6c9e799"
server_key_iv: "d5df6a3009880d860f75e637ba8c76fc"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.053803 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:27.056023 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.057487 2187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:27.057626 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "7cb17aa9843645e1abd0e5ff90dd07bb"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "581f5c398dfc16cb32770ac7e6c9e799"
server_key_iv: "d5df6a3009880d860f75e637ba8c76fc"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.057751 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:27.075608 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:27.078817 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:27.079025 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:27.080569 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:27.080641 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.080705 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:27.080749 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.091815 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44565
I20260927 00:41:27.091871 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44565 every 8 connection(s)
I20260927 00:41:27.092721 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:27.095033 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.081450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38683 (local address 127.0.98.62:35503)
0927 00:41:27.081574 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.081578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.082552 (+ 974us) server_negotiation.cc:407] Connection header received
0927 00:41:27.083526 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.083533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.083601 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.083710 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.085030 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.085800 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.087669 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.088078 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.088195 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.091090 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.091093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.091095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.091119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.092728 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.093372 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.093392 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.093395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.093510 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.093923 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.093926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.093928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.094137 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.094245 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.094723 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.094855 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:27.095731 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.081936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.1:38683)
0927 00:41:27.082410 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.082449 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.083271 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.083866 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.083873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.084257 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.084847 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.084864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.085977 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.085983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.087522 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.087537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.088225 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.088229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.088434 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.089100 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.089115 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.090914 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.092865 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.092870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.092883 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.093249 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.093619 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.093624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.093626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.093830 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.094254 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.094263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.094488 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.094666 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.094955 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260927 00:41:27.097045 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35503
I20260927 00:41:27.097187 2318 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:27.097652 2318 heartbeater.cc:511] Master 127.0.98.62:35503 requested a full tablet report, sending...
I20260927 00:41:27.097858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2187
I20260927 00:41:27.097972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:27.098261 392 external_mini_cluster.cc:1428] Setting key 72357613a7d63ce1185d20edcce3cdb3
I20260927 00:41:27.099200 2114 ts_manager.cc:194] Registered new tserver with Master: 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565)
I20260927 00:41:27.100685 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38683
I20260927 00:41:27.107082 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.099149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.1:44567)
0927 00:41:27.099639 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.099653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.099741 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.099990 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.099993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.100181 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.100328 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.100334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.101449 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.101454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.102014 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.102022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.102563 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.102566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.102637 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.103102 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.103112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.103406 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.105612 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.105614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.105616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.105807 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.106106 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.106109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.106110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.106183 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.106680 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.106683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.106753 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.106849 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.106947 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":90,"threads_started":1}
I20260927 00:41:27.107307 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.099191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44567 (local address 127.0.98.62:35503)
0927 00:41:27.099372 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.099377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.099727 (+ 350us) server_negotiation.cc:407] Connection header received
0927 00:41:27.099825 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.099828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.099871 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.099948 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.100449 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.101293 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.102131 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.102437 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.102555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.103549 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.103556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.103561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.103594 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.105489 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.105911 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.105917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.105919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.106010 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.106323 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.106330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.106335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.106576 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.106690 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.106941 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.107069 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":121}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:27.127331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35503
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:27.233371 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:27.233609 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:27.233652 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:27.237550 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:27.237597 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:27.237634 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:41:27.237649 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:27.237679 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:27.242580 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:27.243403 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:27.244355 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:27.253439 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.253593 2327 server_base.cc:1034] running on GCE node
W20260927 00:41:27.253489 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.253502 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.254066 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:27.254720 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:27.255906 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790469687255896 us; error 25 us; skew 500 ppm
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:27.258646 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:27.259563 2327 webserver.cc:460] Webserver started at http://127.0.98.2:37877/ using document root <none> and password file <none>
I20260927 00:41:27.260056 2327 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:27.260111 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:27.260280 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:27.262477 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbd028650436447d9563e15d18fed935"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "379c87ee998692c66a979bacc2910e42"
server_key_iv: "80cb166c135b178126f54fb7d3f375bf"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.262955 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbd028650436447d9563e15d18fed935"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "379c87ee998692c66a979bacc2910e42"
server_key_iv: "80cb166c135b178126f54fb7d3f375bf"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.266503 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:41:27.268728 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.269974 2327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:27.270063 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "bbd028650436447d9563e15d18fed935"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "379c87ee998692c66a979bacc2910e42"
server_key_iv: "80cb166c135b178126f54fb7d3f375bf"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.270144 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:27.281050 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:27.283759 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:27.283900 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:27.285238 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:27.285288 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.285326 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:27.285385 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.296061 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43543
I20260927 00:41:27.296348 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43543 every 8 connection(s)
I20260927 00:41:27.297272 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:27.299237 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.286042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58901 (local address 127.0.98.62:35503)
0927 00:41:27.286189 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.286195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.287006 (+ 811us) server_negotiation.cc:407] Connection header received
0927 00:41:27.287966 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.287970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.288032 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.288092 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.289742 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.290388 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.291540 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.291841 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.291959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.295115 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.295118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.295121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.295145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.296931 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.297601 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.297608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.297609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.297712 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.298144 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.298168 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.298170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.298401 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.298483 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.299021 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.299122 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:41:27.300030 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.286404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.2:58901)
0927 00:41:27.286829 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.286877 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.287708 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.288596 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.288603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.289005 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.289584 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.289600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.290515 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.290519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.291402 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.291411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.292011 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.292015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.292242 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.292943 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.292968 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.294928 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.297099 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.297107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.297122 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.297513 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.297838 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.297841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.297843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.298031 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.298521 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.298526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.298781 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.298961 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.299208 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
I20260927 00:41:27.301244 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35503
I20260927 00:41:27.301398 2458 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:27.301874 2458 heartbeater.cc:511] Master 127.0.98.62:35503 requested a full tablet report, sending...
I20260927 00:41:27.302850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2327
I20260927 00:41:27.302922 2115 ts_manager.cc:194] Registered new tserver with Master: bbd028650436447d9563e15d18fed935 (127.0.98.2:43543)
I20260927 00:41:27.302973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:27.303270 392 external_mini_cluster.cc:1428] Setting key 1db6adc4b3acb8ec40bdb186e8bb2468
I20260927 00:41:27.303709 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58901
I20260927 00:41:27.311609 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.303471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51075 (local address 127.0.98.62:35503)
0927 00:41:27.303572 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.303576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.303857 (+ 281us) server_negotiation.cc:407] Connection header received
0927 00:41:27.304287 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.304290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.304339 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.304390 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.304982 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.305784 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.306587 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.306815 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.306926 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.308051 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.308057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.308063 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.308094 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.309866 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.310369 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.310373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.310375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.310461 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.310748 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.310751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.310753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.310974 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.311099 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.311359 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.311462 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:41:27.311619 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.303515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.2:51075)
0927 00:41:27.303753 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.303794 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.303894 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.304517 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.304520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.304730 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.304843 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.304849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.305921 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.305923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.306462 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.306471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.306928 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.306931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.307037 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.307585 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.307599 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.307909 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.309978 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.309981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.309982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.310195 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.310557 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.310561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.310563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.310651 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.311086 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.311089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.311190 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.311307 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.311450 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:27.329819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35503
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:27.440599 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:27.440867 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:27.440958 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:27.445102 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:27.445173 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:27.445240 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 00:41:27.445282 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:27.445343 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:27.450426 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:27.451449 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:27.452452 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:27.462342 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:27.462329 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.462716 2467 server_base.cc:1034] running on GCE node
W20260927 00:41:27.462316 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:27.463240 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:27.463805 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:27.464980 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790469687464935 us; error 47 us; skew 500 ppm
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:27.467617 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:27.468487 2467 webserver.cc:460] Webserver started at http://127.0.98.3:38765/ using document root <none> and password file <none>
I20260927 00:41:27.469105 2467 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:27.469193 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:27.469419 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:27.471411 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "56efe16bd4754f95b647a94b81f0038d"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "57ff0dbf08b35107194e1a457890a1e2"
server_key_iv: "bafa6eba937c718dd71134753037745b"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.471966 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56efe16bd4754f95b647a94b81f0038d"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "57ff0dbf08b35107194e1a457890a1e2"
server_key_iv: "bafa6eba937c718dd71134753037745b"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.475620 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:27.478048 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.479178 2467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:27.479310 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "56efe16bd4754f95b647a94b81f0038d"
format_stamp: "Formatted at 2026-09-27 00:41:27 on dist-test-slave-pkws"
server_key: "57ff0dbf08b35107194e1a457890a1e2"
server_key_iv: "bafa6eba937c718dd71134753037745b"
server_key_version: "encryptionkey@0"
I20260927 00:41:27.479416 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:27.509346 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:27.512272 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:27.512497 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:27.513944 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:27.514022 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:27.514096 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:27.514148 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.525028 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38021
I20260927 00:41:27.525090 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38021 every 8 connection(s)
I20260927 00:41:27.525934 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:27.527503 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.514864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36245 (local address 127.0.98.62:35503)
0927 00:41:27.515008 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.515012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.516078 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 00:41:27.517215 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.517219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.517286 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.517390 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.518593 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.519212 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.520359 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.520586 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.520700 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.523845 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.523849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.523852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.523883 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.525500 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.526017 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.526023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.526024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.526116 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.526502 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.526505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.526507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.526723 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.526813 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.527291 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.527390 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:27.528246 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.515382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.3:36245)
0927 00:41:27.515906 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.515948 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.516954 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.517593 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.517599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.517912 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.518437 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.518452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.519343 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.519347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.520207 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.520216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.520734 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.520738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.520987 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.521587 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.521603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.523667 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.525634 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.525640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.525653 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.525913 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.526214 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.526219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.526221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.526390 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.526848 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.526852 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.527062 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.527214 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.527439 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":134,"threads_started":1}
I20260927 00:41:27.529460 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35503
I20260927 00:41:27.529595 2598 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:27.530143 2598 heartbeater.cc:511] Master 127.0.98.62:35503 requested a full tablet report, sending...
I20260927 00:41:27.531179 2115 ts_manager.cc:194] Registered new tserver with Master: 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:27.531852 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36245
I20260927 00:41:27.535902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2467
I20260927 00:41:27.535992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:27.536271 392 external_mini_cluster.cc:1428] Setting key 7dd5279522997b2d3364306f52ba8bc8
I20260927 00:41:27.538532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:27.539162 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.531656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.98.3:38055)
0927 00:41:27.531948 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.531962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.532069 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.532348 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.532351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.532553 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.532646 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.532651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.533618 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.533621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.534156 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.534164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.534594 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.534597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.534674 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.535067 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.535078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.535366 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.537573 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.537575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.537577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.537877 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.538142 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.538145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.538147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.538251 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.538723 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.538727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.538828 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.538921 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.539023 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260927 00:41:27.539291 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.531764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38055 (local address 127.0.98.62:35503)
0927 00:41:27.531880 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.531886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.532044 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:41:27.532161 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.532164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.532211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.532327 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.532757 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.533501 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.534259 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.534497 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.534578 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.535587 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.535590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.535592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.535613 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.537452 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.537970 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.537973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.537974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.538049 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.538404 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.538406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.538407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.538632 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.538725 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.539012 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.539129 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.564121 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.555730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.0.98.62:35503)
0927 00:41:27.555867 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.555871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.555883 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:27.555920 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.555923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.555956 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.556055 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.556736 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.557332 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.558173 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.558404 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.558514 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.561145 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.561148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.561151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.561172 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.562532 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.562956 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.562961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.562963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.563042 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.563308 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.563313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.563314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.563480 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.563629 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.563845 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.563953 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:41:27.566874 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:27.569936 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:27.584339 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.578417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38021 (local address 127.0.0.1:41082)
0927 00:41:27.579342 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.579355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.579445 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.580139 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.580144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.580161 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:27.580473 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.580483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.581708 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.581712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.582853 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.582859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.583704 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.583708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.583850 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.583906 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.584032 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.584162 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260927 00:41:27.584338 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.578973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.0.98.3:38021)
0927 00:41:27.579686 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.579697 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.579720 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:41:27.579791 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.579796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.579993 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.580139 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:27.580609 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.581570 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.583012 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.583598 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.583703 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.583866 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.584097 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.584192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":106,"threads_started":1}
I20260927 00:41:27.584863 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.578290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.0.98.2:43543)
0927 00:41:27.578958 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.578972 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.578996 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:41:27.579077 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.579085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.579302 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.579539 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:27.579854 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.581130 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.582844 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.583986 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.584101 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.584255 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.584609 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.584703 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":113,"threads_started":1}
I20260927 00:41:27.585166 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.578175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43543 (local address 127.0.0.1:41354)
0927 00:41:27.578559 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.578610 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.578733 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.579550 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.579556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.579581 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:27.579716 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.579724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.581250 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.581254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.582687 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.582697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.584330 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.584336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.584451 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.584489 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.584577 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.585026 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260927 00:41:27.585363 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.579343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.0.98.1:44565)
0927 00:41:27.579828 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.579841 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.579859 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:41:27.579918 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.579922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.580069 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.580189 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:27.580741 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.581941 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.583520 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.584353 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.584462 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.584626 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.585103 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.585200 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":112,"threads_started":1}
I20260927 00:41:27.585548 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.578938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.0.1:55978)
0927 00:41:27.579708 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.579721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.579804 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.580316 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.580321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.580338 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:27.580522 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.580528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.582077 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.582080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.583386 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.583397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.584684 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.584688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.584774 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.584795 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.584868 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.584980 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":133,"thread_start_us":101,"threads_started":1}
I20260927 00:41:27.587683 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5774f9822412c87487b8536a4e034 (DEFAULT_TABLE table=test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.588099 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5774f9822412c87487b8536a4e034 (DEFAULT_TABLE table=test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.588097 2237 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5774f9822412c87487b8536a4e034 (DEFAULT_TABLE table=test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.588749 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5774f9822412c87487b8536a4e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.589109 2237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5774f9822412c87487b8536a4e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.589109 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5774f9822412c87487b8536a4e034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.595687 2621 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Bootstrap starting.
I20260927 00:41:27.597151 2622 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Bootstrap starting.
I20260927 00:41:27.598042 2621 tablet_bootstrap.cc:654] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.598845 2621 log.cc:824] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:27.598887 2622 tablet_bootstrap.cc:654] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.599612 2623 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Bootstrap starting.
I20260927 00:41:27.599723 2622 log.cc:824] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:27.601022 2621 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: No bootstrap required, opened a new log
I20260927 00:41:27.601205 2621 ts_tablet_manager.cc:1397] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:41:27.601835 2622 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: No bootstrap required, opened a new log
I20260927 00:41:27.601886 2623 tablet_bootstrap.cc:654] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.602007 2622 ts_tablet_manager.cc:1397] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 00:41:27.602698 2623 log.cc:824] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:27.604315 2623 tablet_bootstrap.cc:492] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: No bootstrap required, opened a new log
I20260927 00:41:27.604511 2623 ts_tablet_manager.cc:1397] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:41:27.605201 2621 raft_consensus.cc:348] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.605480 2621 raft_consensus.cc:374] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.605566 2621 raft_consensus.cc:729] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.605737 2622 raft_consensus.cc:348] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.606041 2622 raft_consensus.cc:374] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.606012 2621 consensus_queue.cc:260] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.606127 2622 raft_consensus.cc:729] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd028650436447d9563e15d18fed935, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.606591 2622 consensus_queue.cc:260] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.607095 2621 ts_tablet_manager.cc:1428] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:41:27.607321 2458 heartbeater.cc:503] Master 127.0.98.62:35503 was elected leader, sending a full tablet report...
I20260927 00:41:27.607528 2598 heartbeater.cc:503] Master 127.0.98.62:35503 was elected leader, sending a full tablet report...
I20260927 00:41:27.608532 2622 ts_tablet_manager.cc:1428] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:27.608775 2623 raft_consensus.cc:348] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.609056 2623 raft_consensus.cc:374] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.609115 2623 raft_consensus.cc:729] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.609543 2623 consensus_queue.cc:260] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.610196 2318 heartbeater.cc:503] Master 127.0.98.62:35503 was elected leader, sending a full tablet report...
I20260927 00:41:27.610499 2623 ts_tablet_manager.cc:1428] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:27.727300 2629 raft_consensus.cc:482] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:27.727545 2629 raft_consensus.cc:504] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.728641 2629 leader_election.cc:288] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:27.732501 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.729235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50909 (local address 127.0.98.2:43543)
0927 00:41:27.729355 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.729359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.729374 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:27.729449 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.729452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.729503 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.729590 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:27.729896 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.730453 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.731485 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.731900 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.732025 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.732088 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.732302 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.732372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:41:27.732523 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.729071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51925 (local address 127.0.98.3:38021)
0927 00:41:27.729208 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.729214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.729363 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:41:27.729529 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.729534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.729579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.729664 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:27.729976 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.730492 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.731528 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.731921 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.732050 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.732111 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.732325 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.732393 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:41:27.732537 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.728879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38021 (local address 127.0.98.1:51925)
0927 00:41:27.729277 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.729293 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.729406 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.729673 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.729676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.729692 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:27.729864 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.729869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.730610 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.730615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.731390 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.731398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.732057 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.732060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.732202 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.732221 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.732308 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.732378 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
I20260927 00:41:27.732553 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.728879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43543 (local address 127.0.98.1:50909)
0927 00:41:27.729206 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.729223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.729354 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.729602 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.729605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.729620 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:27.729758 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.729769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.730568 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.730573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.731358 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.731367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.732035 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.732038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.732137 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.732195 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.732285 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.732362 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":25,"thread_start_us":75,"threads_started":1}
I20260927 00:41:27.733134 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d5774f9822412c87487b8536a4e034" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935" is_pre_election: true
I20260927 00:41:27.733412 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d5774f9822412c87487b8536a4e034" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d" is_pre_election: true
I20260927 00:41:27.733446 2412 raft_consensus.cc:2454] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:27.733752 2552 raft_consensus.cc:2454] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:27.734012 2203 leader_election.cc:302] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:27.734292 2629 raft_consensus.cc:2781] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:27.734378 2629 raft_consensus.cc:482] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:27.734422 2629 raft_consensus.cc:3037] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.735409 2629 raft_consensus.cc:504] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.735735 2629 leader_election.cc:288] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Requested vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:27.736003 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d5774f9822412c87487b8536a4e034" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935"
I20260927 00:41:27.736095 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d5774f9822412c87487b8536a4e034" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d"
I20260927 00:41:27.736131 2412 raft_consensus.cc:3037] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.736250 2552 raft_consensus.cc:3037] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.737268 2412 raft_consensus.cc:2454] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:27.737327 2552 raft_consensus.cc:2454] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:27.737557 2203 leader_election.cc:302] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:27.737735 2629 raft_consensus.cc:2781] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:27.737958 2629 raft_consensus.cc:686] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 LEADER]: Becoming Leader. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Running, Role: LEADER
I20260927 00:41:27.738281 2629 consensus_queue.cc:237] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.741261 2114 catalog_manager.cc:5495] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } health_report { overall_health: UNKNOWN } } }
W20260927 00:41:27.777184 2599 tablet.cc:2369] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:41:27.798473 2459 tablet.cc:2369] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 00:41:27 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:27.803231 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.794588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35503 (local address 127.0.0.1:42820)
0927 00:41:27.794943 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.794958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.795119 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.795424 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.795427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.795888 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:27.795984 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.795991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.796841 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.796850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.797473 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.797482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.797911 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.797915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.797993 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.798636 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:27.798654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:27.800019 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:27.801678 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.801688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.801694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.801926 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.802216 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:27.802219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:27.802221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:27.802303 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:27.802656 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:27.802663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:27.802773 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.802892 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.803043 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
I20260927 00:41:27.803231 2164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.794688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.0.98.62:35503)
0927 00:41:27.794910 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.794916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.795053 (+ 137us) server_negotiation.cc:407] Connection header received
0927 00:41:27.795223 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.795226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.795275 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.795407 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:27.796109 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.796701 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.797580 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.797798 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.797910 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.800147 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:27.800151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:27.800153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:27.800181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:27.801574 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.802044 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.802052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.802058 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.802123 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:27.802395 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:27.802398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:27.802400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:27.802554 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:27.802657 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.802947 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.803094 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":113}
I20260927 00:41:27.807080 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:41:27.807806 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:27.812616 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 0f14b78741a54850b671288eb47cc2f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5875881572004ca7b3c3407c64c06661]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:27.812637 2237 tablet_service.cc:1467] Processing CreateTablet for tablet 0f14b78741a54850b671288eb47cc2f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5875881572004ca7b3c3407c64c06661]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:27.812808 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 0f14b78741a54850b671288eb47cc2f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5875881572004ca7b3c3407c64c06661]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:27.813421 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f14b78741a54850b671288eb47cc2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.813421 2237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f14b78741a54850b671288eb47cc2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.813447 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f14b78741a54850b671288eb47cc2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.816170 2621 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: Bootstrap starting.
I20260927 00:41:27.816344 2622 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: Bootstrap starting.
I20260927 00:41:27.817108 2621 tablet_bootstrap.cc:654] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.817214 2622 tablet_bootstrap.cc:654] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.818044 2623 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: Bootstrap starting.
I20260927 00:41:27.818285 2621 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: No bootstrap required, opened a new log
I20260927 00:41:27.818362 2621 ts_tablet_manager.cc:1397] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:27.818810 2621 raft_consensus.cc:348] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.818895 2621 raft_consensus.cc:374] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.818917 2621 raft_consensus.cc:729] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.818928 2622 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: No bootstrap required, opened a new log
I20260927 00:41:27.818953 2623 tablet_bootstrap.cc:654] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.819001 2622 ts_tablet_manager.cc:1397] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:27.819049 2621 consensus_queue.cc:260] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.819339 2621 ts_tablet_manager.cc:1428] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:27.819314 2627 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.819458 2627 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.819495 2622 raft_consensus.cc:348] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.819603 2622 raft_consensus.cc:374] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.819644 2622 raft_consensus.cc:729] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd028650436447d9563e15d18fed935, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.819773 2622 consensus_queue.cc:260] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.820066 2622 ts_tablet_manager.cc:1428] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:27.820353 2628 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.820488 2628 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.821703 2623 tablet_bootstrap.cc:492] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: No bootstrap required, opened a new log
I20260927 00:41:27.821815 2623 ts_tablet_manager.cc:1397] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:41:27.822346 2623 raft_consensus.cc:348] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.822463 2623 raft_consensus.cc:374] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.822515 2623 raft_consensus.cc:729] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.822664 2623 consensus_queue.cc:260] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.822942 2623 ts_tablet_manager.cc:1428] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:27.822898 2629 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.823027 2629 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 00:41:27.844029 2319 tablet.cc:2369] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:27.863921 2629 raft_consensus.cc:482] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:27.864039 2629 raft_consensus.cc:504] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.864457 2629 leader_election.cc:288] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:27.864928 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f14b78741a54850b671288eb47cc2f9" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935" is_pre_election: true
I20260927 00:41:27.864894 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f14b78741a54850b671288eb47cc2f9" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d" is_pre_election: true
I20260927 00:41:27.865088 2412 raft_consensus.cc:2454] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:27.865118 2552 raft_consensus.cc:2454] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:27.865403 2203 leader_election.cc:302] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:27.865588 2629 raft_consensus.cc:2781] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:27.865649 2629 raft_consensus.cc:482] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:27.865687 2629 raft_consensus.cc:3037] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.866726 2629 raft_consensus.cc:504] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.867041 2629 leader_election.cc:288] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Requested vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:27.867380 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f14b78741a54850b671288eb47cc2f9" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935"
I20260927 00:41:27.867525 2412 raft_consensus.cc:3037] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.867480 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f14b78741a54850b671288eb47cc2f9" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d"
I20260927 00:41:27.867657 2552 raft_consensus.cc:3037] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:27.868468 2412 raft_consensus.cc:2454] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:27.868763 2203 leader_election.cc:302] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:27.868986 2629 raft_consensus.cc:2781] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:27.869001 2552 raft_consensus.cc:2454] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:27.869093 2629 raft_consensus.cc:686] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 LEADER]: Becoming Leader. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Running, Role: LEADER
I20260927 00:41:27.869242 2629 consensus_queue.cc:237] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.869939 2629 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: TxnStatusTablet state changed. Reason: New leader 7cb17aa9843645e1abd0e5ff90dd07bb. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.870051 2629 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:27.870350 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:41:27.870592 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:41:27.871084 2114 catalog_manager.cc:5495] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:27.892043 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.887972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.0.1:55990)
0927 00:41:27.888106 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.888124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.888223 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.888573 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.888575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.888581 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:27.888667 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.888671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.889766 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.889768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.890353 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.890360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.890952 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.890954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.891018 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.891057 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:27.891819 (+ 762us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:27.891829 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.891871 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.891935 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 00:41:27.892119 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.888070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.98.1:44565)
0927 00:41:27.888197 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.888202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.888218 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:27.888368 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.888373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.888445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.888559 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:27.888770 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.889658 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.890465 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.890843 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.890950 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.891190 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:27.891721 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:27.891844 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.891916 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.891975 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:27.897320 2412 raft_consensus.cc:1261] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:27.897320 2552 raft_consensus.cc:1261] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:27.898017 2629 consensus_queue.cc:1035] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:27.898249 2632 consensus_queue.cc:1035] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:27.901753 2628 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: TxnStatusTablet state changed. Reason: New leader 7cb17aa9843645e1abd0e5ff90dd07bb. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.901863 2628 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.903563 2627 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: TxnStatusTablet state changed. Reason: New leader 7cb17aa9843645e1abd0e5ff90dd07bb. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.903684 2627 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.905913 2629 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.906029 2629 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:27.906138 2629 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.906201 2629 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 7cb17aa9843645e1abd0e5ff90dd07bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:27.906291 2647 mvcc.cc:205] Tried to move back new op lower bound from 7333763841620029440 to 7333763841513754624. Current Snapshot: MvccSnapshot[applied={T|T < 7333763841620029440}]
I20260927 00:41:27.906585 2648 mvcc.cc:205] Tried to move back new op lower bound from 7333763841620029440 to 7333763841513754624. Current Snapshot: MvccSnapshot[applied={T|T < 7333763841620029440}]
I20260927 00:41:27.906935 2627 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.907035 2627 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P 56efe16bd4754f95b647a94b81f0038d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.907114 2650 mvcc.cc:205] Tried to move back new op lower bound from 7333763841620029440 to 7333763841513754624. Current Snapshot: MvccSnapshot[applied={T|T < 7333763841620029440}]
I20260927 00:41:27.907881 2628 tablet_replica.cc:439] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } } }
I20260927 00:41:27.907980 2628 tablet_replica.cc:442] T 0f14b78741a54850b671288eb47cc2f9 P bbd028650436447d9563e15d18fed935: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:27.917089 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.913705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.0.98.1:44565)
0927 00:41:27.913829 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.913835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.913849 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:27.914006 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.914009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.914068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.914144 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:27.914408 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.914909 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.915893 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.916125 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.916266 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.916577 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:27.916641 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:27.916724 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.916836 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.916877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:27.927440 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.924311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35259 (local address 127.0.98.1:44565)
0927 00:41:27.924423 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:27.924427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:27.924586 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:41:27.924731 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:27.924736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:27.924779 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:27.924855 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:27.925108 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.925579 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.926411 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.926623 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.926730 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.927009 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:27.927075 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:27.927153 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:27.927234 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 00:41:27.927284 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260927 00:41:27.927455 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:27.924241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.98.1:35259)
0927 00:41:27.924507 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:27.924519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:27.924633 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:27.924864 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:27.924866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:27.924873 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:27.924999 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.925004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.925685 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.925690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.926297 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:27.926305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:27.926731 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:27.926734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:27.926826 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:27.926869 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:27.927156 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:27.927163 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:41:27.927227 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 00:41:27.927306 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260927 00:41:27.933924 2412 raft_consensus.cc:1261] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:27.934043 2552 raft_consensus.cc:1261] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:27.934342 2632 consensus_queue.cc:1035] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:27.934492 2652 consensus_queue.cc:1035] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:27.975100 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:27.975291 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:27.977676 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Sending DeleteTablet for 3 replicas of tablet 04d5774f9822412c87487b8536a4e034
I20260927 00:41:27.978403 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d5774f9822412c87487b8536a4e034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 00:41:27.978454 2238 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d5774f9822412c87487b8536a4e034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 00:41:27.978533 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d5774f9822412c87487b8536a4e034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41354
I20260927 00:41:27.978991 2663 tablet_replica.cc:330] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: stopping tablet replica
I20260927 00:41:27.979023 2664 tablet_replica.cc:330] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: stopping tablet replica
I20260927 00:41:27.979326 2664 raft_consensus.cc:2229] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:27.979327 2663 raft_consensus.cc:2229] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:27.979532 2663 raft_consensus.cc:2258] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:27.979554 2664 raft_consensus.cc:2258] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:27.979594 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:41:27.980019 2665 tablet_replica.cc:330] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: stopping tablet replica
W20260927 00:41:27.980187 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:27.980834 2665 raft_consensus.cc:2229] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:27.981244 2665 raft_consensus.cc:2258] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:27.981257 2664 ts_tablet_manager.cc:1905] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:27.982586 2665 ts_tablet_manager.cc:1905] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:27.984351 2664 ts_tablet_manager.cc:1918] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:27.985328 2663 ts_tablet_manager.cc:1905] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:27.985363 2238 tablet_service.cc:1467] Processing CreateTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 (DEFAULT_TABLE table=test-table [id=c97a4898fc6949118578d32c797ac715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.985628 2238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5966eb3f9d4f00967714473f6a4b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.986054 2532 tablet_service.cc:1467] Processing CreateTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 (DEFAULT_TABLE table=test-table [id=c97a4898fc6949118578d32c797ac715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.986265 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5966eb3f9d4f00967714473f6a4b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.988129 2663 ts_tablet_manager.cc:1918] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:27.988207 2663 log.cc:1196] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/04d5774f9822412c87487b8536a4e034
I20260927 00:41:27.988320 2664 log.cc:1196] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/04d5774f9822412c87487b8536a4e034
I20260927 00:41:27.988436 2623 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Bootstrap starting.
I20260927 00:41:27.988669 2664 ts_tablet_manager.cc:1939] T 04d5774f9822412c87487b8536a4e034 P bbd028650436447d9563e15d18fed935: Deleting consensus metadata
I20260927 00:41:27.989404 2623 tablet_bootstrap.cc:654] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.989974 2099 catalog_manager.cc:4841] TS bbd028650436447d9563e15d18fed935 (127.0.98.2:43543): tablet 04d5774f9822412c87487b8536a4e034 (table test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]) successfully deleted
I20260927 00:41:27.990031 2621 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Bootstrap starting.
I20260927 00:41:27.990129 2663 ts_tablet_manager.cc:1939] T 04d5774f9822412c87487b8536a4e034 P 56efe16bd4754f95b647a94b81f0038d: Deleting consensus metadata
I20260927 00:41:27.990876 2621 tablet_bootstrap.cc:654] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.985538 2392 tablet_service.cc:1467] Processing CreateTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 (DEFAULT_TABLE table=test-table [id=c97a4898fc6949118578d32c797ac715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:27.992092 2621 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: No bootstrap required, opened a new log
I20260927 00:41:27.992111 2100 catalog_manager.cc:4841] TS 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021): tablet 04d5774f9822412c87487b8536a4e034 (table test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]) successfully deleted
I20260927 00:41:27.992164 2621 ts_tablet_manager.cc:1397] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:27.992185 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd5966eb3f9d4f00967714473f6a4b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:27.992583 2621 raft_consensus.cc:348] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.992681 2621 raft_consensus.cc:374] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.992709 2621 raft_consensus.cc:729] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.992823 2621 consensus_queue.cc:260] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.993089 2621 ts_tablet_manager.cc:1428] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:27.993609 2623 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: No bootstrap required, opened a new log
I20260927 00:41:27.993680 2623 ts_tablet_manager.cc:1397] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260927 00:41:27.994093 2623 raft_consensus.cc:348] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.994231 2623 raft_consensus.cc:374] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.994266 2623 raft_consensus.cc:729] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.994377 2623 consensus_queue.cc:260] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.994475 2622 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Bootstrap starting.
I20260927 00:41:27.994640 2623 ts_tablet_manager.cc:1428] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:27.995425 2622 tablet_bootstrap.cc:654] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:27.996673 2622 tablet_bootstrap.cc:492] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: No bootstrap required, opened a new log
I20260927 00:41:27.996778 2622 ts_tablet_manager.cc:1397] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:27.997231 2622 raft_consensus.cc:348] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.997359 2622 raft_consensus.cc:374] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:27.997395 2622 raft_consensus.cc:729] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd028650436447d9563e15d18fed935, State: Initialized, Role: FOLLOWER
I20260927 00:41:27.997514 2622 consensus_queue.cc:260] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:27.997774 2622 ts_tablet_manager.cc:1428] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:28.001376 2665 ts_tablet_manager.cc:1918] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:28.001453 2665 log.cc:1196] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/04d5774f9822412c87487b8536a4e034
I20260927 00:41:28.001803 2665 ts_tablet_manager.cc:1939] T 04d5774f9822412c87487b8536a4e034 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting consensus metadata
I20260927 00:41:28.002915 2099 catalog_manager.cc:4841] TS 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565): tablet 04d5774f9822412c87487b8536a4e034 (table test-table [id=0bde2b4d1b7a4612b558b6c70f93297b]) successfully deleted
I20260927 00:41:28.187883 2652 raft_consensus.cc:482] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:28.188045 2652 raft_consensus.cc:504] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:28.188511 2652 leader_election.cc:288] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:28.188953 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5966eb3f9d4f00967714473f6a4b01" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935" is_pre_election: true
I20260927 00:41:28.188953 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5966eb3f9d4f00967714473f6a4b01" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d" is_pre_election: true
I20260927 00:41:28.189098 2412 raft_consensus.cc:2454] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:28.189126 2552 raft_consensus.cc:2454] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 0.
I20260927 00:41:28.189389 2204 leader_election.cc:302] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56efe16bd4754f95b647a94b81f0038d, 7cb17aa9843645e1abd0e5ff90dd07bb; no voters:
I20260927 00:41:28.189600 2652 raft_consensus.cc:2781] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:28.189661 2652 raft_consensus.cc:482] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:28.189687 2652 raft_consensus.cc:3037] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:28.190652 2652 raft_consensus.cc:504] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:28.191002 2652 leader_election.cc:288] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Requested vote from peers bbd028650436447d9563e15d18fed935 (127.0.98.2:43543), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:28.191324 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5966eb3f9d4f00967714473f6a4b01" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935"
I20260927 00:41:28.191330 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd5966eb3f9d4f00967714473f6a4b01" candidate_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d"
I20260927 00:41:28.191465 2552 raft_consensus.cc:3037] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:28.191464 2412 raft_consensus.cc:3037] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:28.192304 2552 raft_consensus.cc:2454] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:28.192406 2412 raft_consensus.cc:2454] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cb17aa9843645e1abd0e5ff90dd07bb in term 1.
I20260927 00:41:28.192564 2204 leader_election.cc:302] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56efe16bd4754f95b647a94b81f0038d, 7cb17aa9843645e1abd0e5ff90dd07bb; no voters:
I20260927 00:41:28.192718 2652 raft_consensus.cc:2781] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:28.192824 2652 raft_consensus.cc:686] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 LEADER]: Becoming Leader. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Running, Role: LEADER
I20260927 00:41:28.193007 2652 consensus_queue.cc:237] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:28.194550 2113 catalog_manager.cc:5495] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:28.246285 2412 raft_consensus.cc:1261] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:28.246287 2552 raft_consensus.cc:1261] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Refusing update from remote peer 7cb17aa9843645e1abd0e5ff90dd07bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:28.246652 2652 consensus_queue.cc:1035] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:28.246830 2632 consensus_queue.cc:1035] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:41:28.259832 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:41:29.807457 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:29.807615 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:29.809998 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Sending DeleteTablet for 3 replicas of tablet bd5966eb3f9d4f00967714473f6a4b01
I20260927 00:41:29.810520 2238 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 00:41:29.810621 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41354
I20260927 00:41:29.810871 2692 tablet_replica.cc:330] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: stopping tablet replica
I20260927 00:41:29.810982 2693 tablet_replica.cc:330] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: stopping tablet replica
I20260927 00:41:29.811040 2692 raft_consensus.cc:2229] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:29.811136 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet bd5966eb3f9d4f00967714473f6a4b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 00:41:29.811182 2693 raft_consensus.cc:2229] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:29.811262 2692 raft_consensus.cc:2258] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:29.811303 2693 raft_consensus.cc:2258] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:29.812036 2692 ts_tablet_manager.cc:1905] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:29.812386 2693 ts_tablet_manager.cc:1905] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:29.814388 2694 tablet_replica.cc:330] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: stopping tablet replica
I20260927 00:41:29.814517 2694 raft_consensus.cc:2229] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:29.814663 2694 raft_consensus.cc:2258] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:29.815140 2692 ts_tablet_manager.cc:1918] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:41:29.815222 2692 log.cc:1196] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/bd5966eb3f9d4f00967714473f6a4b01
I20260927 00:41:29.815333 2694 ts_tablet_manager.cc:1905] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:29.815609 2693 ts_tablet_manager.cc:1918] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:41:29.815690 2693 log.cc:1196] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/bd5966eb3f9d4f00967714473f6a4b01
I20260927 00:41:29.815950 2693 ts_tablet_manager.cc:1939] T bd5966eb3f9d4f00967714473f6a4b01 P bbd028650436447d9563e15d18fed935: Deleting consensus metadata
I20260927 00:41:29.816633 2099 catalog_manager.cc:4841] TS bbd028650436447d9563e15d18fed935 (127.0.98.2:43543): tablet bd5966eb3f9d4f00967714473f6a4b01 (table test-table [id=c97a4898fc6949118578d32c797ac715]) successfully deleted
I20260927 00:41:29.817232 2692 ts_tablet_manager.cc:1939] T bd5966eb3f9d4f00967714473f6a4b01 P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting consensus metadata
I20260927 00:41:29.817838 2099 catalog_manager.cc:4841] TS 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565): tablet bd5966eb3f9d4f00967714473f6a4b01 (table test-table [id=c97a4898fc6949118578d32c797ac715]) successfully deleted
I20260927 00:41:29.818008 2694 ts_tablet_manager.cc:1918] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:41:29.818078 2694 log.cc:1196] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/bd5966eb3f9d4f00967714473f6a4b01
I20260927 00:41:29.818321 2694 ts_tablet_manager.cc:1939] T bd5966eb3f9d4f00967714473f6a4b01 P 56efe16bd4754f95b647a94b81f0038d: Deleting consensus metadata
I20260927 00:41:29.818960 2100 catalog_manager.cc:4841] TS 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021): tablet bd5966eb3f9d4f00967714473f6a4b01 (table test-table [id=c97a4898fc6949118578d32c797ac715]) successfully deleted
Sep 27 00:41:29 dist-test-slave-pkws krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:41:29 dist-test-slave-pkws krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469689, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:29.847716 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:29.833228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.0.98.62:35503)
0927 00:41:29.834511 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:29.834521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:29.834534 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:29.834575 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:29.834577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:29.834615 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:29.834692 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:29.835356 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.836356 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.837312 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:29.837547 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:29.837661 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:29.844558 (+ 6897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:29.844561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:29.844563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:29.844588 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:29.846057 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.846505 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.846507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.846509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.846585 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:29.846840 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:29.846842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:29.846843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:29.847077 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:29.847210 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:29.847400 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 00:41:29.847500 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":116,"threads_started":1}
I20260927 00:41:29.850410 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:29.850888 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:29.855494 2238 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb83e770854d7791db1c2905839c3a (DEFAULT_TABLE table=test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:29.855506 2392 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb83e770854d7791db1c2905839c3a (DEFAULT_TABLE table=test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:29.855600 2532 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb83e770854d7791db1c2905839c3a (DEFAULT_TABLE table=test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:29.855803 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb83e770854d7791db1c2905839c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:29.855799 2238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb83e770854d7791db1c2905839c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:29.855823 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb83e770854d7791db1c2905839c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:29.858484 2705 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Bootstrap starting.
I20260927 00:41:29.858500 2703 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Bootstrap starting.
I20260927 00:41:29.859269 2705 tablet_bootstrap.cc:654] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:29.859292 2703 tablet_bootstrap.cc:654] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:29.859947 2704 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Bootstrap starting.
I20260927 00:41:29.860491 2705 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: No bootstrap required, opened a new log
I20260927 00:41:29.860575 2705 ts_tablet_manager.cc:1397] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:29.860836 2704 tablet_bootstrap.cc:654] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:29.861169 2705 raft_consensus.cc:348] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.861303 2705 raft_consensus.cc:374] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:29.861342 2705 raft_consensus.cc:729] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Initialized, Role: FOLLOWER
I20260927 00:41:29.861512 2705 consensus_queue.cc:260] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.861912 2705 ts_tablet_manager.cc:1428] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:41:29.862509 2704 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: No bootstrap required, opened a new log
I20260927 00:41:29.862536 2703 tablet_bootstrap.cc:492] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: No bootstrap required, opened a new log
I20260927 00:41:29.862620 2703 ts_tablet_manager.cc:1397] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 00:41:29.862627 2704 ts_tablet_manager.cc:1397] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:29.863015 2704 raft_consensus.cc:348] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.863054 2703 raft_consensus.cc:348] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.863127 2704 raft_consensus.cc:374] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:29.863164 2704 raft_consensus.cc:729] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd028650436447d9563e15d18fed935, State: Initialized, Role: FOLLOWER
I20260927 00:41:29.863170 2703 raft_consensus.cc:374] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:29.863227 2703 raft_consensus.cc:729] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Initialized, Role: FOLLOWER
I20260927 00:41:29.863322 2704 consensus_queue.cc:260] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.863365 2703 consensus_queue.cc:260] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:29.863667 2703 ts_tablet_manager.cc:1428] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:29.863698 2704 ts_tablet_manager.cc:1428] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:30.233184 2708 raft_consensus.cc:482] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:30.233314 2708 raft_consensus.cc:504] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.234416 2708 leader_election.cc:288] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565), bbd028650436447d9563e15d18fed935 (127.0.98.2:43543)
I20260927 00:41:30.239959 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.235324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58729 (local address 127.0.98.2:43543)
0927 00:41:30.235620 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.235628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.235641 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:30.235807 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.235809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.235865 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.235949 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:30.236203 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.236748 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.238511 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.239361 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.239492 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.239589 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.239749 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.239827 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260927 00:41:30.239959 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.235158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43543 (local address 127.0.98.3:58729)
0927 00:41:30.235544 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.235560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.235680 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.235951 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.235954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.235976 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:30.236091 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.236096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.236862 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.236868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.238361 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.238372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.239507 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.239511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.239594 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.239620 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.239719 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.239798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":66,"thread_start_us":86,"threads_started":1}
I20260927 00:41:30.240458 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb83e770854d7791db1c2905839c3a" candidate_uuid: "56efe16bd4754f95b647a94b81f0038d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935" is_pre_election: true
I20260927 00:41:30.240694 2412 raft_consensus.cc:2454] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56efe16bd4754f95b647a94b81f0038d in term 0.
I20260927 00:41:30.241084 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.234710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.98.3:37463)
0927 00:41:30.235052 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.235068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.235188 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.235589 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.235593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.235613 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:30.235792 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.235799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.237055 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.237058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.238368 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.238374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.240574 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.240577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.240705 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.240724 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.240815 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.240920 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":74,"spinlock_wait_cycles":39808,"thread_start_us":140,"threads_started":1}
I20260927 00:41:30.241178 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.234800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37463 (local address 127.0.98.1:44565)
0927 00:41:30.235095 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.235110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.235131 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:30.235322 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.235325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.235375 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.235457 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:30.235935 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.236916 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.239656 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.240439 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.240568 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.240631 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.240846 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.240974 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260927 00:41:30.241155 2483 leader_election.cc:302] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56efe16bd4754f95b647a94b81f0038d, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:30.241436 2708 raft_consensus.cc:2781] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:30.241499 2708 raft_consensus.cc:482] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:30.241551 2708 raft_consensus.cc:3037] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.241942 2258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb83e770854d7791db1c2905839c3a" candidate_uuid: "56efe16bd4754f95b647a94b81f0038d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" is_pre_election: true
I20260927 00:41:30.242179 2258 raft_consensus.cc:2454] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56efe16bd4754f95b647a94b81f0038d in term 0.
I20260927 00:41:30.243142 2708 raft_consensus.cc:504] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.243577 2708 leader_election.cc:288] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [CANDIDATE]: Term 1 election: Requested vote from peers 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565), bbd028650436447d9563e15d18fed935 (127.0.98.2:43543)
I20260927 00:41:30.243963 2258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb83e770854d7791db1c2905839c3a" candidate_uuid: "56efe16bd4754f95b647a94b81f0038d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb"
I20260927 00:41:30.243990 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb83e770854d7791db1c2905839c3a" candidate_uuid: "56efe16bd4754f95b647a94b81f0038d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd028650436447d9563e15d18fed935"
I20260927 00:41:30.244094 2258 raft_consensus.cc:3037] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.244143 2412 raft_consensus.cc:3037] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.245092 2258 raft_consensus.cc:2454] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56efe16bd4754f95b647a94b81f0038d in term 1.
I20260927 00:41:30.245357 2483 leader_election.cc:302] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56efe16bd4754f95b647a94b81f0038d, 7cb17aa9843645e1abd0e5ff90dd07bb; no voters:
I20260927 00:41:30.245536 2412 raft_consensus.cc:2454] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56efe16bd4754f95b647a94b81f0038d in term 1.
I20260927 00:41:30.245582 2708 raft_consensus.cc:2781] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:30.245787 2708 raft_consensus.cc:686] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 1 LEADER]: Becoming Leader. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Running, Role: LEADER
I20260927 00:41:30.246054 2708 consensus_queue.cc:237] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.248659 2114 catalog_manager.cc:5495] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d reported cstate change: term changed from 0 to 1, leader changed from <none> to 56efe16bd4754f95b647a94b81f0038d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "56efe16bd4754f95b647a94b81f0038d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:30.316794 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.313013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.98.3:38021)
0927 00:41:30.313335 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.313348 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.313365 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:41:30.313452 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.313455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.313518 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.313622 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.313839 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.314374 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.315236 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.315474 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.315550 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.315886 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.316358 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.316442 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.316527 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.316589 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":136,"threads_started":1}
I20260927 00:41:30.325672 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.322834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40881 (local address 127.0.98.1:44565)
0927 00:41:30.322952 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.322955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.323113 (+ 158us) server_negotiation.cc:407] Connection header received
0927 00:41:30.323188 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.323224 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.323266 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.323327 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.323509 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.323983 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.324751 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.325022 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.325098 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.325335 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.325395 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.325446 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.325519 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.325556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:30.325685 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.322765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.98.3:40881)
0927 00:41:30.323033 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.323045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.323126 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.323348 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.323350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.323356 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:30.323427 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.323431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.324065 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.324067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.324657 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.324665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.325122 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.325125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.325190 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.325232 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.325464 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.325468 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.325514 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.325573 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260927 00:41:30.331000 2412 raft_consensus.cc:1261] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Refusing update from remote peer 56efe16bd4754f95b647a94b81f0038d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:30.331084 2258 raft_consensus.cc:1261] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Refusing update from remote peer 56efe16bd4754f95b647a94b81f0038d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:30.331516 2708 consensus_queue.cc:1035] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:30.331691 2715 consensus_queue.cc:1035] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:30.349025 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.345771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48553 (local address 127.0.98.3:38021)
0927 00:41:30.345881 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.345887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.346102 (+ 215us) server_negotiation.cc:407] Connection header received
0927 00:41:30.346186 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.346188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.346225 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.346283 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.346433 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.346928 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.347800 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.348164 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.348285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.348528 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.348612 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.348688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.348787 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.348843 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:41:30.349037 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.345683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38021 (local address 127.0.98.1:48553)
0927 00:41:30.346029 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.346041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.346123 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.346286 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.346288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.346293 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:30.346373 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.346377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.347010 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.347013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.347683 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.347695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.348274 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.348276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.348370 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.348413 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.348713 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.348718 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.348773 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.348869 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260927 00:41:30.371914 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:30.372056 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:30.374120 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Sending DeleteTablet for 3 replicas of tablet dbeb83e770854d7791db1c2905839c3a
I20260927 00:41:30.374715 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb83e770854d7791db1c2905839c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41354
I20260927 00:41:30.374755 2238 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb83e770854d7791db1c2905839c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 00:41:30.374763 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb83e770854d7791db1c2905839c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 00:41:30.375054 2736 tablet_replica.cc:330] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: stopping tablet replica
I20260927 00:41:30.375133 2738 tablet_replica.cc:330] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: stopping tablet replica
I20260927 00:41:30.375185 2736 raft_consensus.cc:2229] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:30.375245 2738 raft_consensus.cc:2229] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:30.375294 2736 raft_consensus.cc:2258] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:30.375340 2738 raft_consensus.cc:2258] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:30.375557 2737 tablet_replica.cc:330] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: stopping tablet replica
I20260927 00:41:30.375685 2737 raft_consensus.cc:2229] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:30.375950 2736 ts_tablet_manager.cc:1905] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:30.375952 2737 raft_consensus.cc:2258] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:30.375874 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:30.376257 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:30.376516 2737 ts_tablet_manager.cc:1905] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:30.378026 2738 ts_tablet_manager.cc:1905] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:30.380302 2736 ts_tablet_manager.cc:1918] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:30.380380 2736 log.cc:1196] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/dbeb83e770854d7791db1c2905839c3a
I20260927 00:41:30.380666 2736 ts_tablet_manager.cc:1939] T dbeb83e770854d7791db1c2905839c3a P bbd028650436447d9563e15d18fed935: Deleting consensus metadata
I20260927 00:41:30.380648 2738 ts_tablet_manager.cc:1918] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:30.380715 2738 log.cc:1196] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/dbeb83e770854d7791db1c2905839c3a
I20260927 00:41:30.380977 2738 ts_tablet_manager.cc:1939] T dbeb83e770854d7791db1c2905839c3a P 7cb17aa9843645e1abd0e5ff90dd07bb: Deleting consensus metadata
I20260927 00:41:30.381332 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb10dff286b4c48a791252db1b61175 (DEFAULT_TABLE table=test-table [id=aad69c5d88944f4b974700b7d75e9c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:30.381400 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb10dff286b4c48a791252db1b61175 (DEFAULT_TABLE table=test-table [id=aad69c5d88944f4b974700b7d75e9c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:30.381572 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb10dff286b4c48a791252db1b61175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:30.381595 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb10dff286b4c48a791252db1b61175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:30.381624 2238 tablet_service.cc:1467] Processing CreateTablet for tablet 3bb10dff286b4c48a791252db1b61175 (DEFAULT_TABLE table=test-table [id=aad69c5d88944f4b974700b7d75e9c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:30.381852 2238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bb10dff286b4c48a791252db1b61175. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:30.384411 2099 catalog_manager.cc:4841] TS bbd028650436447d9563e15d18fed935 (127.0.98.2:43543): tablet dbeb83e770854d7791db1c2905839c3a (table test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]) successfully deleted
I20260927 00:41:30.384676 2099 catalog_manager.cc:4841] TS 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565): tablet dbeb83e770854d7791db1c2905839c3a (table test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]) successfully deleted
I20260927 00:41:30.384730 2741 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: Bootstrap starting.
I20260927 00:41:30.385581 2741 tablet_bootstrap.cc:654] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:30.385857 2737 ts_tablet_manager.cc:1918] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:30.385932 2737 log.cc:1196] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/dbeb83e770854d7791db1c2905839c3a
I20260927 00:41:30.386209 2739 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Bootstrap starting.
I20260927 00:41:30.386204 2737 ts_tablet_manager.cc:1939] T dbeb83e770854d7791db1c2905839c3a P 56efe16bd4754f95b647a94b81f0038d: Deleting consensus metadata
I20260927 00:41:30.386843 2100 catalog_manager.cc:4841] TS 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021): tablet dbeb83e770854d7791db1c2905839c3a (table test-table [id=c72961dcf3d744d6a8c57f0d89451a9a]) successfully deleted
I20260927 00:41:30.387135 2739 tablet_bootstrap.cc:654] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:30.388051 2741 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: No bootstrap required, opened a new log
I20260927 00:41:30.388144 2741 ts_tablet_manager.cc:1397] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:30.388343 2739 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: No bootstrap required, opened a new log
I20260927 00:41:30.388424 2739 ts_tablet_manager.cc:1397] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:30.388566 2741 raft_consensus.cc:348] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.388682 2741 raft_consensus.cc:374] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:30.388734 2741 raft_consensus.cc:729] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb17aa9843645e1abd0e5ff90dd07bb, State: Initialized, Role: FOLLOWER
I20260927 00:41:30.388845 2739 raft_consensus.cc:348] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.388868 2741 consensus_queue.cc:260] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.388985 2739 raft_consensus.cc:374] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:30.389027 2739 raft_consensus.cc:729] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd028650436447d9563e15d18fed935, State: Initialized, Role: FOLLOWER
I20260927 00:41:30.389201 2740 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Bootstrap starting.
I20260927 00:41:30.389179 2739 consensus_queue.cc:260] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.390156 2739 ts_tablet_manager.cc:1428] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:41:30.390604 2740 tablet_bootstrap.cc:654] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:30.391002 2741 ts_tablet_manager.cc:1428] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:30.397218 2740 tablet_bootstrap.cc:492] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: No bootstrap required, opened a new log
I20260927 00:41:30.397306 2740 ts_tablet_manager.cc:1397] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 00:41:30.397795 2740 raft_consensus.cc:348] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.397912 2740 raft_consensus.cc:374] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:30.397943 2740 raft_consensus.cc:729] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56efe16bd4754f95b647a94b81f0038d, State: Initialized, Role: FOLLOWER
I20260927 00:41:30.398061 2740 consensus_queue.cc:260] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.398326 2740 ts_tablet_manager.cc:1428] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:30.439482 2710 raft_consensus.cc:482] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:30.439636 2710 raft_consensus.cc:504] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.440732 2710 leader_election.cc:288] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:30.445606 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.441238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59997 (local address 127.0.98.1:44565)
0927 00:41:30.441360 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.441364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.441473 (+ 109us) server_negotiation.cc:407] Connection header received
0927 00:41:30.441631 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.441634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.441682 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.441748 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:30.442092 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.442653 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.444153 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.444768 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.444871 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.444961 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.445404 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.445480 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:41:30.445606 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.441100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49505 (local address 127.0.98.3:38021)
0927 00:41:30.441232 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.441236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.441507 (+ 271us) server_negotiation.cc:407] Connection header received
0927 00:41:30.441729 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.441734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.441788 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.441855 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:30.442201 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.442966 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.444321 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.444740 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.444858 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.444944 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.445404 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.445480 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:41:30.445971 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.440929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.98.2:59997)
0927 00:41:30.441369 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.441384 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.441506 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.441768 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.441771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.441789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:30.441958 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.441966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.442770 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.442772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.444026 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.444036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.445011 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.445017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.445231 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.445274 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.445739 (+ 465us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.445825 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":10,"thread_start_us":124,"threads_started":1}
I20260927 00:41:30.446305 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.441054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38021 (local address 127.0.98.2:49505)
0927 00:41:30.441429 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.441442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.441565 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.441971 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.441974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.441991 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:30.442107 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.442114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.443096 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.443099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.444206 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.444213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.444869 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.444872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.445227 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.445275 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.445997 (+ 722us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.446149 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":86,"thread_start_us":108,"threads_started":1}
I20260927 00:41:30.446431 2258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb10dff286b4c48a791252db1b61175" candidate_uuid: "bbd028650436447d9563e15d18fed935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" is_pre_election: true
I20260927 00:41:30.446583 2258 raft_consensus.cc:2454] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd028650436447d9563e15d18fed935 in term 0.
I20260927 00:41:30.446739 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb10dff286b4c48a791252db1b61175" candidate_uuid: "bbd028650436447d9563e15d18fed935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d" is_pre_election: true
I20260927 00:41:30.446864 2552 raft_consensus.cc:2454] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd028650436447d9563e15d18fed935 in term 0.
I20260927 00:41:30.447266 2344 leader_election.cc:302] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56efe16bd4754f95b647a94b81f0038d, 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:30.447537 2710 raft_consensus.cc:2781] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:30.447595 2710 raft_consensus.cc:482] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:30.447639 2710 raft_consensus.cc:3037] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.448575 2710 raft_consensus.cc:504] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.449026 2710 leader_election.cc:288] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [CANDIDATE]: Term 1 election: Requested vote from peers 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565), 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021)
I20260927 00:41:30.449301 2258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb10dff286b4c48a791252db1b61175" candidate_uuid: "bbd028650436447d9563e15d18fed935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb"
I20260927 00:41:30.449352 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bb10dff286b4c48a791252db1b61175" candidate_uuid: "bbd028650436447d9563e15d18fed935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56efe16bd4754f95b647a94b81f0038d"
I20260927 00:41:30.449435 2258 raft_consensus.cc:3037] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.449467 2552 raft_consensus.cc:3037] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:30.450344 2258 raft_consensus.cc:2454] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd028650436447d9563e15d18fed935 in term 1.
I20260927 00:41:30.450555 2552 raft_consensus.cc:2454] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd028650436447d9563e15d18fed935 in term 1.
I20260927 00:41:30.450647 2343 leader_election.cc:302] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cb17aa9843645e1abd0e5ff90dd07bb, bbd028650436447d9563e15d18fed935; no voters:
I20260927 00:41:30.450824 2710 raft_consensus.cc:2781] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:30.451026 2710 raft_consensus.cc:686] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 1 LEADER]: Becoming Leader. State: Replica: bbd028650436447d9563e15d18fed935, State: Running, Role: LEADER
I20260927 00:41:30.451283 2710 consensus_queue.cc:237] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } }
I20260927 00:41:30.453557 2114 catalog_manager.cc:5495] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbd028650436447d9563e15d18fed935 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bbd028650436447d9563e15d18fed935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbd028650436447d9563e15d18fed935" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:30.473196 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.469433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.0.98.2:43543)
0927 00:41:30.469553 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.469557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.469572 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:30.469722 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.469725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.469787 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.469894 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.470061 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.470899 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.471698 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.472021 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.472129 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.472331 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.472803 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.472936 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.473007 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.473061 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:30.484180 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.479923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50199 (local address 127.0.98.1:44565)
0927 00:41:30.480078 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.480082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.480206 (+ 124us) server_negotiation.cc:407] Connection header received
0927 00:41:30.480320 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.480322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.480361 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.480423 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.480705 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.481247 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.483095 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.483320 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.483394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.483733 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.483793 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.483841 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.484006 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.484066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 00:41:30.484192 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.479772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.98.2:50199)
0927 00:41:30.480122 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.480136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.480238 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.480452 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.480455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.480464 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:30.480599 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.480606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.481345 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.481348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.482975 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.482990 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.483450 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.483455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.483558 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.483605 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.483872 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.483879 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.483948 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.484021 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
I20260927 00:41:30.490106 2552 raft_consensus.cc:1261] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Refusing update from remote peer bbd028650436447d9563e15d18fed935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:30.490209 2258 raft_consensus.cc:1261] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Refusing update from remote peer bbd028650436447d9563e15d18fed935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:30.490631 2747 consensus_queue.cc:1035] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56efe16bd4754f95b647a94b81f0038d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:30.490893 2710 consensus_queue.cc:1035] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cb17aa9843645e1abd0e5ff90dd07bb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:30.508750 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.504765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43543 (local address 127.0.98.1:46857)
0927 00:41:30.504891 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:30.504921 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:30.505000 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:30.505413 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:30.505416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:30.505424 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:30.505516 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.505523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.506438 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.506441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.507358 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:30.507370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.507859 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.507863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:30.507950 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.507985 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.508440 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.508447 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:41:30.508523 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:30.508616 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 00:41:30.509191 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:30.504995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46857 (local address 127.0.98.2:43543)
0927 00:41:30.505125 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:30.505130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:30.505141 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:41:30.505189 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:30.505192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:30.505236 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:30.505305 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:30.505646 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.506326 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.507495 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:30.507728 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:30.507818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:30.508239 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:30.508323 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:30.508394 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:30.509028 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 00:41:30.509080 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 00:41:30.509874 2720 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 00:41:32.081590 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:32.081869 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:32.084582 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d3786261fa4fadb2d8ca38a6323ab5: Sending DeleteTablet for 3 replicas of tablet 3bb10dff286b4c48a791252db1b61175
I20260927 00:41:32.085136 2238 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bb10dff286b4c48a791252db1b61175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 00:41:32.085232 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bb10dff286b4c48a791252db1b61175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 00:41:32.085315 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bb10dff286b4c48a791252db1b61175 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41354
I20260927 00:41:32.085471 2780 tablet_replica.cc:330] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb: stopping tablet replica
I20260927 00:41:32.085544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2187
I20260927 00:41:32.085604 2780 raft_consensus.cc:2229] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:32.085716 2780 raft_consensus.cc:2258] T 3bb10dff286b4c48a791252db1b61175 P 7cb17aa9843645e1abd0e5ff90dd07bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:32.086891 2781 tablet_replica.cc:330] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: stopping tablet replica
I20260927 00:41:32.087383 2781 raft_consensus.cc:2229] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:32.087502 2781 raft_consensus.cc:2258] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:32.087847 2782 tablet_replica.cc:330] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: stopping tablet replica
I20260927 00:41:32.088006 2782 raft_consensus.cc:2229] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:32.088207 2781 ts_tablet_manager.cc:1905] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:32.088330 2782 raft_consensus.cc:2258] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:32.089026 2782 ts_tablet_manager.cc:1905] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:32.090557 2781 ts_tablet_manager.cc:1918] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:41:32.090644 2781 log.cc:1196] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/3bb10dff286b4c48a791252db1b61175
I20260927 00:41:32.090927 2781 ts_tablet_manager.cc:1939] T 3bb10dff286b4c48a791252db1b61175 P 56efe16bd4754f95b647a94b81f0038d: Deleting consensus metadata
I20260927 00:41:32.091675 2100 catalog_manager.cc:4841] TS 56efe16bd4754f95b647a94b81f0038d (127.0.98.3:38021): tablet 3bb10dff286b4c48a791252db1b61175 (table test-table [id=aad69c5d88944f4b974700b7d75e9c74]) successfully deleted
I20260927 00:41:32.092543 2782 ts_tablet_manager.cc:1918] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 00:41:32.092629 2782 log.cc:1196] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/3bb10dff286b4c48a791252db1b61175
I20260927 00:41:32.092941 2782 ts_tablet_manager.cc:1939] T 3bb10dff286b4c48a791252db1b61175 P bbd028650436447d9563e15d18fed935: Deleting consensus metadata
I20260927 00:41:32.093708 2099 catalog_manager.cc:4841] TS bbd028650436447d9563e15d18fed935 (127.0.98.2:43543): tablet 3bb10dff286b4c48a791252db1b61175 (table test-table [id=aad69c5d88944f4b974700b7d75e9c74]) successfully deleted
W20260927 00:41:32.095108 2099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44565 (error 108)
I20260927 00:41:32.095348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2327
I20260927 00:41:32.096556 2784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:32.096180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44565 (local address 127.0.0.1:56014)
0927 00:41:32.096408 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:32.096480 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
W20260927 00:41:32.096688 2099 catalog_manager.cc:4584] TS 7cb17aa9843645e1abd0e5ff90dd07bb (127.0.98.1:44565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bb10dff286b4c48a791252db1b61175: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44565: connect: Connection refused (error 111)
I20260927 00:41:32.103085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2467
I20260927 00:41:32.109380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2083
2026-09-27T00:41:32Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6603 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:32 dist-test-slave-pkws krb5kdc[2787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:32 dist-test-slave-pkws krb5kdc[2787](info): set up 2 sockets
Sep 27 00:41:32 dist-test-slave-pkws krb5kdc[2787](info): commencing operation
krb5kdc: starting...
W20260927 00:41:33.144759 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:33.260411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:33.366036 2800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:33.366305 2800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:33.366372 2800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:33.370354 2800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:33.370434 2800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:33.370468 2800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:33.370503 2800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:33.370538 2800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:33.375612 2800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:33.376622 2800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:33.377619 2800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:33.386443 2806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:33.386427 2808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:33.386427 2805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:33.386791 2800 server_base.cc:1034] running on GCE node
I20260927 00:41:33.387328 2800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:33.388378 2800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:33.389551 2800 hybrid_clock.cc:648] HybridClock initialized: now 1790469693389533 us; error 31 us; skew 500 ppm
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:33.392318 2800 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:33.393452 2800 webserver.cc:460] Webserver started at http://127.0.98.62:36587/ using document root <none> and password file <none>
I20260927 00:41:33.394033 2800 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:33.394096 2800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:33.394317 2800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:33.396423 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "fa986159eb5b4258bb4926413d6a3c1f"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "24e2d67a80951bee6e9205f6caa08d70"
server_key_iv: "e481a2171d74005d8d4e38bb6393ada0"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.397042 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa986159eb5b4258bb4926413d6a3c1f"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "24e2d67a80951bee6e9205f6caa08d70"
server_key_iv: "e481a2171d74005d8d4e38bb6393ada0"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.400542 2800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:33.402974 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.404145 2800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:33.404294 2800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "fa986159eb5b4258bb4926413d6a3c1f"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "24e2d67a80951bee6e9205f6caa08d70"
server_key_iv: "e481a2171d74005d8d4e38bb6393ada0"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.404393 2800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:33.421028 2800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:33.424244 2800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:33.424446 2800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:33.431097 2800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36159
I20260927 00:41:33.431165 2867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36159 every 8 connection(s)
I20260927 00:41:33.432054 2800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:33.434752 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:33.435608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2800
I20260927 00:41:33.435693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:33.435957 392 external_mini_cluster.cc:1428] Setting key 0ec8fc50aabf31c444b82fdce08aa75a
I20260927 00:41:33.441015 2868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f: Bootstrap starting.
I20260927 00:41:33.443054 2868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:33.443735 2868 log.cc:824] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:33.445679 2868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f: No bootstrap required, opened a new log
I20260927 00:41:33.449038 2868 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 00:41:33.449273 2868 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:33.449349 2868 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa986159eb5b4258bb4926413d6a3c1f, State: Initialized, Role: FOLLOWER
I20260927 00:41:33.449596 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:33.437296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.0.98.62:36159)
0927 00:41:33.437766 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:33.437787 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:33.437831 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:41:33.438510 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:33.438527 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:33.438836 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:33.439101 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:33.439967 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.441024 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.441878 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.442352 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.442516 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:33.445038 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:33.445049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:33.445061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:33.445085 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:33.446750 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.447202 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.447208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.447220 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.447331 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.447614 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.447619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.447621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.447973 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:33.448193 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:33.448430 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 00:41:33.448687 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":109,"threads_started":1}
I20260927 00:41:33.449837 2868 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 00:41:33.449987 2868 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:33.450074 2868 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:33.450179 2868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:33.451215 2868 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 00:41:33.451558 2868 leader_election.cc:302] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa986159eb5b4258bb4926413d6a3c1f; no voters:
I20260927 00:41:33.451894 2868 leader_election.cc:288] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:33.451978 2873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:33.452167 2873 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [term 1 LEADER]: Becoming Leader. State: Replica: fa986159eb5b4258bb4926413d6a3c1f, State: Running, Role: LEADER
I20260927 00:41:33.452437 2873 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } }
I20260927 00:41:33.453140 2868 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:33.453615 2875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa986159eb5b4258bb4926413d6a3c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } } }
I20260927 00:41:33.453644 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa986159eb5b4258bb4926413d6a3c1f. Latest consensus state: current_term: 1 leader_uuid: "fa986159eb5b4258bb4926413d6a3c1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa986159eb5b4258bb4926413d6a3c1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36159 } } }
I20260927 00:41:33.453794 2875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:33.453831 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:33.454290 2878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:33.457274 2878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:33.462630 2878 catalog_manager.cc:1347] Generated new cluster ID: efd5fc378a654873b1c5811a7dd1776d
I20260927 00:41:33.462692 2878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:33.471161 2878 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:33.471984 2878 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:33.477262 2878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa986159eb5b4258bb4926413d6a3c1f: Generated new TSK 0
I20260927 00:41:33.477833 2878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:33.507090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:33.615402 2894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:33.615625 2894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:33.615666 2894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:33.619741 2894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:33.619793 2894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:33.619884 2894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:33.625085 2894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:33.626034 2894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:33.626984 2894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:33.636562 2900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:33.636564 2899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:33.636627 2894 server_base.cc:1034] running on GCE node
W20260927 00:41:33.636641 2902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:33.637157 2894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:33.637758 2894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:33.638943 2894 hybrid_clock.cc:648] HybridClock initialized: now 1790469693638927 us; error 28 us; skew 500 ppm
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:33.641563 2894 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:33.642406 2894 webserver.cc:460] Webserver started at http://127.0.98.1:40851/ using document root <none> and password file <none>
I20260927 00:41:33.642927 2894 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:33.643021 2894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:33.643241 2894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:33.645233 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c93220e5e5a4e108c2993bfa1fd35a7"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "a1fba75bea6eceddf9380c6c90d476a9"
server_key_iv: "68cb2990d17078b24bc83e31598f9d33"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.645879 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c93220e5e5a4e108c2993bfa1fd35a7"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "a1fba75bea6eceddf9380c6c90d476a9"
server_key_iv: "68cb2990d17078b24bc83e31598f9d33"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.649326 2894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:33.651671 2909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.653000 2894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:33.653118 2894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "3c93220e5e5a4e108c2993bfa1fd35a7"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "a1fba75bea6eceddf9380c6c90d476a9"
server_key_iv: "68cb2990d17078b24bc83e31598f9d33"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.653234 2894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:33.663718 2894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:33.666425 2894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:33.666613 2894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:33.667140 2894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:33.667882 2894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:33.667953 2894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.668028 2894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:33.668063 2894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.676896 2894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39777
I20260927 00:41:33.676992 3022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39777 every 8 connection(s)
I20260927 00:41:33.677781 2894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:33.682691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 2894
I20260927 00:41:33.682782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:33.683039 392 external_mini_cluster.cc:1428] Setting key 8bd18d71c044e4f7d3122646bafe5c83
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:33.691377 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:33.679482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35859 (local address 127.0.98.62:36159)
0927 00:41:33.679602 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:33.679606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:33.680388 (+ 782us) server_negotiation.cc:407] Connection header received
0927 00:41:33.681848 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:33.681852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:33.681910 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:33.681985 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:33.683131 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.683670 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.684793 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.685093 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.685202 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:33.687830 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:33.687834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:33.687836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:33.687857 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:33.689398 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.689904 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.689908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.689910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.689997 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.690332 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.690337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.690339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.690516 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:33.690625 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:33.691086 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 00:41:33.691205 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:33.692368 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:33.679815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36159 (local address 127.0.98.1:35859)
0927 00:41:33.680230 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:33.680289 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:33.681630 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:33.682147 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:33.682153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:33.682487 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:33.682988 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:33.683003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.683801 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.683804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:33.684659 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:33.684668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.685246 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.685250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:33.685447 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:33.686034 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:33.686060 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:33.687664 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:33.689532 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:33.689538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:33.689551 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:33.689799 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:33.690098 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:33.690101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:33.690103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:33.690235 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:33.690622 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:33.690627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:33.690860 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 00:41:33.691046 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:41:33.691343 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260927 00:41:33.693584 3023 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36159
I20260927 00:41:33.693862 3023 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:33.694481 3023 heartbeater.cc:511] Master 127.0.98.62:36159 requested a full tablet report, sending...
I20260927 00:41:33.696002 2832 ts_manager.cc:194] Registered new tserver with Master: 3c93220e5e5a4e108c2993bfa1fd35a7 (127.0.98.1:39777)
I20260927 00:41:33.697172 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:33.708288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:33.813853 3028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:33.814088 3028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:33.814127 3028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:33.818038 3028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:33.818087 3028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:33.818142 3028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:33.822993 3028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:33.823868 3028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:33.825081 3028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:33.840382 3036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:33.840431 3034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:33.840476 3028 server_base.cc:1034] running on GCE node
W20260927 00:41:33.840391 3033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:33.841049 3028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:33.841610 3028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:33.842767 3028 hybrid_clock.cc:648] HybridClock initialized: now 1790469693842748 us; error 31 us; skew 500 ppm
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:33.845503 3028 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:33.846271 3028 webserver.cc:460] Webserver started at http://127.0.98.2:41455/ using document root <none> and password file <none>
I20260927 00:41:33.846776 3028 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:33.846833 3028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:33.846998 3028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:33.848891 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "caaba7041510411f8cc8d9483cbebbab"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "d1b050774ed664bd1c15b09f20caf4cc"
server_key_iv: "fa5e93b6e672409577ca47010d73ddb8"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.849488 3028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "caaba7041510411f8cc8d9483cbebbab"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "d1b050774ed664bd1c15b09f20caf4cc"
server_key_iv: "fa5e93b6e672409577ca47010d73ddb8"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.853048 3028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:33.855571 3043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.856781 3028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:33.856900 3028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "caaba7041510411f8cc8d9483cbebbab"
format_stamp: "Formatted at 2026-09-27 00:41:33 on dist-test-slave-pkws"
server_key: "d1b050774ed664bd1c15b09f20caf4cc"
server_key_iv: "fa5e93b6e672409577ca47010d73ddb8"
server_key_version: "encryptionkey@0"
I20260927 00:41:33.857038 3028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:33.881021 3028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:33.884272 3028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:33.884424 3028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:33.884994 3028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:33.885753 3028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:33.885800 3028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.885838 3028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:33.885859 3028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:33.894413 3028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36181
I20260927 00:41:33.894443 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36181 every 8 connection(s)
I20260927 00:41:33.895268 3028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:33.904196 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3028
I20260927 00:41:33.904309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
Sep 27 00:41:33 dist-test-slave-pkws krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:33.904637 392 external_mini_cluster.cc:1428] Setting key fb9a7a5d64fc4e97363f9ab50ae0dee6
I20260927 00:41:33.909250 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:33.896991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40411 (local address 127.0.98.62:36159)
0927 00:41:33.897124 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:33.897127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:33.897930 (+ 803us) server_negotiation.cc:407] Connection header received
0927 00:41:33.899289 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:33.899293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:33.899349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:33.899458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:33.900570 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.901217 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.902334 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.902650 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.902765 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:33.905360 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:33.905364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:33.905366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:33.905388 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:33.907074 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.907635 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.907641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.907646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.907759 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:33.908156 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:33.908158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:33.908159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:33.908329 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:33.908404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:33.909031 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 00:41:33.909127 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:41:33.910542 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:33.897363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36159 (local address 127.0.98.2:40411)
0927 00:41:33.897784 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:33.897825 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:33.899083 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:33.899638 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:33.899645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:33.899975 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:33.900417 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:33.900430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.901341 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.901344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:33.902204 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:33.902213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:33.902798 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:33.902802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:33.902966 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:33.903464 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:33.903490 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:33.905168 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:33.907220 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:33.907224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:33.907235 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:33.907491 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:33.907863 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:33.907868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:33.907870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:33.908060 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:33.908429 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:33.908437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:33.908755 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 00:41:33.909003 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 00:41:33.909442 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260927 00:41:33.911755 3157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36159
I20260927 00:41:33.911998 3157 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:33.912468 3157 heartbeater.cc:511] Master 127.0.98.62:36159 requested a full tablet report, sending...
I20260927 00:41:33.913501 2832 ts_manager.cc:194] Registered new tserver with Master: caaba7041510411f8cc8d9483cbebbab (127.0.98.2:36181)
I20260927 00:41:33.914052 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40411
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:33.931934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:34.040354 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:34.040628 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:34.040707 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:34.044718 3162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:34.044808 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:34.044950 3162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:34.050081 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:34.051081 3162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:34.052044 3162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:34.061802 3170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.061802 3167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:34.061853 3168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:34.062129 3162 server_base.cc:1034] running on GCE node
I20260927 00:41:34.062568 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:34.063169 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:34.064352 3162 hybrid_clock.cc:648] HybridClock initialized: now 1790469694064339 us; error 29 us; skew 500 ppm
Sep 27 00:41:34 dist-test-slave-pkws krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:34.067147 3162 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:34.067981 3162 webserver.cc:460] Webserver started at http://127.0.98.3:44841/ using document root <none> and password file <none>
I20260927 00:41:34.068523 3162 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:34.068614 3162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:34.068835 3162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:34.070771 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6db4a2c149c4dd7891ccb394cca6119"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-pkws"
server_key: "3a15d7de3e0fbcd970748fd314437eea"
server_key_iv: "3afa3a41ee5b7de850612bfb0d7def6e"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.071329 3162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6db4a2c149c4dd7891ccb394cca6119"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-pkws"
server_key: "3a15d7de3e0fbcd970748fd314437eea"
server_key_iv: "3afa3a41ee5b7de850612bfb0d7def6e"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.074786 3162 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:34.077078 3177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.078230 3162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:34.078361 3162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b6db4a2c149c4dd7891ccb394cca6119"
format_stamp: "Formatted at 2026-09-27 00:41:34 on dist-test-slave-pkws"
server_key: "3a15d7de3e0fbcd970748fd314437eea"
server_key_iv: "3afa3a41ee5b7de850612bfb0d7def6e"
server_key_version: "encryptionkey@0"
I20260927 00:41:34.078464 3162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:34.108839 3162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:34.112104 3162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:34.112303 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:34.112861 3162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:34.113701 3162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:34.113773 3162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.113842 3162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:34.113895 3162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:34.122740 3162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32881
I20260927 00:41:34.122768 3290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32881 every 8 connection(s)
I20260927 00:41:34.123564 3162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:34.127918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3162
I20260927 00:41:34.128037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:34.128309 392 external_mini_cluster.cc:1428] Setting key 103ffdf4142596f35a5ea5f93e6954c0
Sep 27 00:41:34 dist-test-slave-pkws krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:34.136549 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.125232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47949 (local address 127.0.98.62:36159)
0927 00:41:34.125394 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.125398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.126157 (+ 759us) server_negotiation.cc:407] Connection header received
0927 00:41:34.127436 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.127439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.127482 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.127556 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.128761 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.129583 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.130950 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.131152 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.131231 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.133439 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:34.133442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:34.133444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:34.133466 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:34.134897 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.135329 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.135332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.135333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.135410 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:34.135688 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:34.135690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:34.135691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:34.135867 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:34.135939 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:34.136347 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 00:41:34.136434 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:41:34.137362 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:34.125603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36159 (local address 127.0.98.3:47949)
0927 00:41:34.126013 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:34.126054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:34.127252 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:34.127712 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:34.127718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:34.128084 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:34.128623 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.128637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.129721 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.129723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.130846 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:34.130856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:34.131271 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:34.131274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:34.131402 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:34.131893 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:34.131908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:34.133301 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:34.135000 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.135004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.135015 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.135240 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.135489 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:34.135492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:34.135494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:34.135612 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:34.135948 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:34.135953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:34.136160 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 00:41:34.136304 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 00:41:34.136524 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260927 00:41:34.138438 3291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36159
I20260927 00:41:34.138665 3291 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:34.139117 3291 heartbeater.cc:511] Master 127.0.98.62:36159 requested a full tablet report, sending...
I20260927 00:41:34.140074 2832 ts_manager.cc:194] Registered new tserver with Master: b6db4a2c149c4dd7891ccb394cca6119 (127.0.98.3:32881)
I20260927 00:41:34.140703 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47949
I20260927 00:41:34.142128 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:41:34.154582 2871 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:34.152967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.0.98.62:36159)
0927 00:41:34.153088 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:34.153092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:34.153343 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:41:34.153512 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:34.153516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:34.153564 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:34.153645 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:34.154417 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45806: BlockingRecv error: recv got EOF from 127.0.0.1:45806 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:41:34.155524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2894
I20260927 00:41:34.161082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3028
I20260927 00:41:34.166049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3162
I20260927 00:41:34.170897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 2800
2026-09-27T00:41:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2059 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:34 dist-test-slave-pkws krb5kdc[3303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:34 dist-test-slave-pkws krb5kdc[3303](info): set up 2 sockets
Sep 27 00:41:34 dist-test-slave-pkws krb5kdc[3303](info): commencing operation
krb5kdc: starting...
W20260927 00:41:35.204790 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:35.319705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:35.430112 3316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:35.430389 3316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:35.430454 3316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:35.434423 3316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:35.434495 3316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:35.434535 3316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:35.434588 3316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:35.434628 3316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:35.439677 3316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:35.440636 3316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:35.441613 3316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:35.449579 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:35.449607 3321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:35.449580 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:35.449795 3316 server_base.cc:1034] running on GCE node
I20260927 00:41:35.450376 3316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:35.451473 3316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:35.452666 3316 hybrid_clock.cc:648] HybridClock initialized: now 1790469695452655 us; error 28 us; skew 500 ppm
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:35.462339 3316 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:35.463182 3316 webserver.cc:460] Webserver started at http://127.0.98.62:35693/ using document root <none> and password file <none>
I20260927 00:41:35.463697 3316 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:35.463753 3316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:35.463912 3316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:35.465948 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "0eed31fa96fd4cfabe18d936fdc4163c"
format_stamp: "Formatted at 2026-09-27 00:41:35 on dist-test-slave-pkws"
server_key: "0f40d2957c0a2a1ae033094e732594e8"
server_key_iv: "95e4a1421d172f1198f7931bc6ae993a"
server_key_version: "encryptionkey@0"
I20260927 00:41:35.466466 3316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "0eed31fa96fd4cfabe18d936fdc4163c"
format_stamp: "Formatted at 2026-09-27 00:41:35 on dist-test-slave-pkws"
server_key: "0f40d2957c0a2a1ae033094e732594e8"
server_key_iv: "95e4a1421d172f1198f7931bc6ae993a"
server_key_version: "encryptionkey@0"
I20260927 00:41:35.470048 3316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:41:35.472250 3331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:35.473546 3316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:35.473646 3316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "0eed31fa96fd4cfabe18d936fdc4163c"
format_stamp: "Formatted at 2026-09-27 00:41:35 on dist-test-slave-pkws"
server_key: "0f40d2957c0a2a1ae033094e732594e8"
server_key_iv: "95e4a1421d172f1198f7931bc6ae993a"
server_key_version: "encryptionkey@0"
I20260927 00:41:35.473726 3316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:35.487720 3316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:35.490294 3316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:35.490437 3316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:35.496847 3316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41619
I20260927 00:41:35.497231 3383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41619 every 8 connection(s)
I20260927 00:41:35.497921 3316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:35.500543 3384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:35.505112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3316
I20260927 00:41:35.505224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:35.505494 392 external_mini_cluster.cc:1428] Setting key 256af8bf56200030ca192364590fbec2
I20260927 00:41:35.505988 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c: Bootstrap starting.
I20260927 00:41:35.508198 3384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:35.509145 3384 log.cc:824] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:35.511052 3384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c: No bootstrap required, opened a new log
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:35.515287 3384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260927 00:41:35.515556 3384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:35.515646 3384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eed31fa96fd4cfabe18d936fdc4163c, State: Initialized, Role: FOLLOWER
I20260927 00:41:35.516124 3384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260927 00:41:35.516273 3384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:35.516340 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:35.516453 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:35.517308 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260927 00:41:35.517660 3384 leader_election.cc:302] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eed31fa96fd4cfabe18d936fdc4163c; no voters:
I20260927 00:41:35.517961 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:35.518056 3389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:35.518280 3389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [term 1 LEADER]: Becoming Leader. State: Replica: 0eed31fa96fd4cfabe18d936fdc4163c, State: Running, Role: LEADER
I20260927 00:41:35.518689 3389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } }
I20260927 00:41:35.519109 3384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:35.519909 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eed31fa96fd4cfabe18d936fdc4163c. Latest consensus state: current_term: 1 leader_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } } }
I20260927 00:41:35.520107 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:35.520350 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:35.506805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.98.62:41619)
0927 00:41:35.507200 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:35.507215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:35.507248 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:41:35.507843 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:35.507859 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:35.508179 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:35.508487 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:35.509441 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:35.510412 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:35.511559 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:35.512026 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:35.512148 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:35.514856 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:35.514861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:35.514874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:35.514900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:35.516753 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:35.517230 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:35.517235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:35.517242 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:35.517355 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:35.517675 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:35.517677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:35.517678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:35.518213 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:35.518961 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:35.519123 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:41:35.519390 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260927 00:41:35.520373 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eed31fa96fd4cfabe18d936fdc4163c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41619 } } }
I20260927 00:41:35.520462 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:35.521618 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:35.523299 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:35.524745 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:35.530457 3398 catalog_manager.cc:1347] Generated new cluster ID: 322bc0afabca4ba7a723dbc2feb7100b
I20260927 00:41:35.530510 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:35.541796 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:35.536521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.0.98.62:41619)
0927 00:41:35.536625 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:35.536630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:35.536838 (+ 208us) server_negotiation.cc:407] Connection header received
0927 00:41:35.536986 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:35.536989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:35.537035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:35.537136 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:35.537650 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:35.538437 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:35.539514 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:35.539899 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:35.539996 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:35.540825 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:35.540829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:35.540832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:35.541389 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:35.541421 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:35.541441 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:35.541532 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:35.541612 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 00:41:35.541664 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 00:41:35.548184 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:35.549754 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:35.555209 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eed31fa96fd4cfabe18d936fdc4163c: Generated new TSK 0
I20260927 00:41:35.555928 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:35.564395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3316
2026-09-27T00:41:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1392 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3417](info): set up 2 sockets
Sep 27 00:41:35 dist-test-slave-pkws krb5kdc[3417](info): commencing operation
krb5kdc: starting...
W20260927 00:41:36.620846 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:36 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:36.736289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36825
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:36.842941 3430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:36.843223 3430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:36.843300 3430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:36.848573 3430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:36.848668 3430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:36.848711 3430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:36.848747 3430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:36.848788 3430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:36.856295 3430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:36.857427 3430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:36.858392 3430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:36.866645 3438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:36.866632 3435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:36.866627 3436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:36.866847 3430 server_base.cc:1034] running on GCE node
I20260927 00:41:36.867496 3430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:36.868624 3430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:36.869779 3430 hybrid_clock.cc:648] HybridClock initialized: now 1790469696869757 us; error 34 us; skew 500 ppm
Sep 27 00:41:36 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:36.872430 3430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:36.873378 3430 webserver.cc:460] Webserver started at http://127.0.98.62:37557/ using document root <none> and password file <none>
I20260927 00:41:36.873881 3430 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:36.873939 3430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:36.874101 3430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:36.876044 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "e7f9c3d609234bbdaa3f2760b831cac4"
format_stamp: "Formatted at 2026-09-27 00:41:36 on dist-test-slave-pkws"
server_key: "880ee92227b4cffe533b026dca672f7d"
server_key_iv: "4b9b29f006eb57ff70decbac9c745bf8"
server_key_version: "encryptionkey@0"
I20260927 00:41:36.876560 3430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7f9c3d609234bbdaa3f2760b831cac4"
format_stamp: "Formatted at 2026-09-27 00:41:36 on dist-test-slave-pkws"
server_key: "880ee92227b4cffe533b026dca672f7d"
server_key_iv: "4b9b29f006eb57ff70decbac9c745bf8"
server_key_version: "encryptionkey@0"
I20260927 00:41:36.880137 3430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:36.882421 3445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:36.883615 3430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:36.883741 3430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "e7f9c3d609234bbdaa3f2760b831cac4"
format_stamp: "Formatted at 2026-09-27 00:41:36 on dist-test-slave-pkws"
server_key: "880ee92227b4cffe533b026dca672f7d"
server_key_iv: "4b9b29f006eb57ff70decbac9c745bf8"
server_key_version: "encryptionkey@0"
I20260927 00:41:36.883843 3430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:36.897153 3430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:36.899776 3430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:36.899964 3430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:36.906965 3430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36825
I20260927 00:41:36.907020 3497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36825 every 8 connection(s)
I20260927 00:41:36.907881 3430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:36.910596 3498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:36.911626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3430
I20260927 00:41:36.911734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:36.912042 392 external_mini_cluster.cc:1428] Setting key a224c3080d9ee5d479112847e04d0557
I20260927 00:41:36.916791 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: Bootstrap starting.
I20260927 00:41:36.918905 3498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:36.919588 3498 log.cc:824] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:41:36 dist-test-slave-pkws krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:36.921571 3498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: No bootstrap required, opened a new log
I20260927 00:41:36.925102 3498 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } }
I20260927 00:41:36.925335 3498 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:36.925415 3498 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f9c3d609234bbdaa3f2760b831cac4, State: Initialized, Role: FOLLOWER
I20260927 00:41:36.925451 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:36.913355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.0.98.62:36825)
0927 00:41:36.913747 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:36.913761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:36.913794 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:41:36.914400 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:36.914416 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:36.914732 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:36.914997 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:36.915880 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:36.916833 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:36.917721 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:36.918193 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:36.918374 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:36.920849 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:36.920854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:36.920867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:36.920942 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:36.922616 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:36.923088 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:36.923093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:36.923097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:36.923212 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:36.923492 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:36.923495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:36.923497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:36.923867 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:36.924076 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:36.924340 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 00:41:36.924611 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260927 00:41:36.925905 3498 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } }
I20260927 00:41:36.926059 3498 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:36.926119 3498 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:36.926208 3498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:36.927094 3498 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } }
I20260927 00:41:36.927450 3498 leader_election.cc:302] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7f9c3d609234bbdaa3f2760b831cac4; no voters:
I20260927 00:41:36.927788 3498 leader_election.cc:288] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:36.927898 3503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:36.928113 3503 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [term 1 LEADER]: Becoming Leader. State: Replica: e7f9c3d609234bbdaa3f2760b831cac4, State: Running, Role: LEADER
I20260927 00:41:36.928447 3503 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } }
I20260927 00:41:36.928946 3498 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:36.930548 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } } }
I20260927 00:41:36.930688 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:36.930850 3505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7f9c3d609234bbdaa3f2760b831cac4. Latest consensus state: current_term: 1 leader_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7f9c3d609234bbdaa3f2760b831cac4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36825 } } }
I20260927 00:41:36.931385 3505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4 [sys.catalog]: This master's current role is: LEADER
W20260927 00:41:36.933501 3517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:36.933564 3517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:36.933638 3519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:36.934412 3519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:36.939052 3519 catalog_manager.cc:1347] Generated new cluster ID: f887e6c7010d48638760886dda322bf0
I20260927 00:41:36.939111 3519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:36.954054 3519 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:36.954952 3519 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:36.963851 3519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7f9c3d609234bbdaa3f2760b831cac4: Generated new TSK 0
I20260927 00:41:36.964445 3519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:36.990123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:37.097752 3524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:37.098011 3524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:37.098086 3524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:37.102011 3524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:37.102088 3524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:37.102171 3524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:37.107154 3524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:37.108125 3524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:37.109141 3524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:37.117995 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.118029 3524 server_base.cc:1034] running on GCE node
W20260927 00:41:37.117981 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.117970 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.118557 3524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:37.119140 3524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:37.120293 3524 hybrid_clock.cc:648] HybridClock initialized: now 1790469697120279 us; error 26 us; skew 500 ppm
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:37.122948 3524 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:37.123692 3524 webserver.cc:460] Webserver started at http://127.0.98.1:36761/ using document root <none> and password file <none>
I20260927 00:41:37.124171 3524 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:37.124228 3524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:37.124377 3524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:37.126317 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed71f265d1204e96a08c083ae8a3ace2"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "a05ec8e4f18c5226413c8d5fd9e6c4ad"
server_key_iv: "5b47d1653956dc8d97b63dfbac1b3646"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.126814 3524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed71f265d1204e96a08c083ae8a3ace2"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "a05ec8e4f18c5226413c8d5fd9e6c4ad"
server_key_iv: "5b47d1653956dc8d97b63dfbac1b3646"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.130373 3524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:37.132604 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.133790 3524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:37.133912 3524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "ed71f265d1204e96a08c083ae8a3ace2"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "a05ec8e4f18c5226413c8d5fd9e6c4ad"
server_key_iv: "5b47d1653956dc8d97b63dfbac1b3646"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.134022 3524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:37.155527 3524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:37.158320 3524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:37.158514 3524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:37.159063 3524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:37.159848 3524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:37.159916 3524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.159998 3524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:37.160050 3524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.169119 3524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40979
I20260927 00:41:37.169152 3652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40979 every 8 connection(s)
I20260927 00:41:37.169960 3524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:37.175999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3524
I20260927 00:41:37.176115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:37.176429 392 external_mini_cluster.cc:1428] Setting key 8a74e2cedba6780c6b16a775f3ccee87
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:37.184859 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.171653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41241 (local address 127.0.98.62:36825)
0927 00:41:37.171780 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:37.171784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:37.172542 (+ 758us) server_negotiation.cc:407] Connection header received
0927 00:41:37.173888 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:37.173891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:37.173942 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:37.174036 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:37.175170 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.175700 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.177390 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.177743 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.177855 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.180804 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:37.180811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:37.180817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:37.180848 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:37.182669 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.183224 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.183230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.183234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.183333 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.183672 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.183677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.183682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.183938 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:37.184055 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:37.184515 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 00:41:37.184682 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:41:37.185572 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.171993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36825 (local address 127.0.98.1:41241)
0927 00:41:37.172395 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:37.172435 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:37.173672 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:37.174220 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:37.174226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:37.174578 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:37.175035 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.175049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.175841 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.175846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.177279 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.177289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.177894 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.177897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.178126 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.178718 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:37.178744 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:37.180646 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:37.182782 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.182787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.182797 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.183117 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.183427 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.183429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.183430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.183579 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.184059 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:37.184065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:37.184289 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 00:41:37.184439 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:41:37.184697 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:37.186827 3653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36825
I20260927 00:41:37.187079 3653 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:37.187489 3653 heartbeater.cc:511] Master 127.0.98.62:36825 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:37.189131 3462 ts_manager.cc:194] Registered new tserver with Master: ed71f265d1204e96a08c083ae8a3ace2 (127.0.98.1:40979)
I20260927 00:41:37.190248 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:37.203830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:37.307690 3658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:37.307934 3658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:37.307976 3658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:37.311911 3658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:37.311962 3658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:37.312057 3658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:37.317106 3658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:37.318050 3658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:37.319013 3658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:37.328331 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.328323 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.328323 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.328459 3658 server_base.cc:1034] running on GCE node
I20260927 00:41:37.329103 3658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:37.329814 3658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:37.331009 3658 hybrid_clock.cc:648] HybridClock initialized: now 1790469697330983 us; error 45 us; skew 500 ppm
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:37.333782 3658 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:37.334652 3658 webserver.cc:460] Webserver started at http://127.0.98.2:41063/ using document root <none> and password file <none>
I20260927 00:41:37.335224 3658 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:37.335317 3658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:37.335543 3658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:37.337817 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7715f7847f24718b72dab3fcd6b85a4"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "5c5634ea09850b704e95b4f0bb3daa52"
server_key_iv: "04fd76d1e8c0759a88a1a30df32aecea"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.338402 3658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7715f7847f24718b72dab3fcd6b85a4"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "5c5634ea09850b704e95b4f0bb3daa52"
server_key_iv: "04fd76d1e8c0759a88a1a30df32aecea"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.342108 3658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:37.344610 3673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.345827 3658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:37.345960 3658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "d7715f7847f24718b72dab3fcd6b85a4"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "5c5634ea09850b704e95b4f0bb3daa52"
server_key_iv: "04fd76d1e8c0759a88a1a30df32aecea"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.346104 3658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:37.379963 3658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:37.383076 3658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:37.383278 3658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:37.383899 3658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:37.384733 3658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:37.384801 3658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.384869 3658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:37.384934 3658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.393959 3658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40327
I20260927 00:41:37.393983 3786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40327 every 8 connection(s)
I20260927 00:41:37.394930 3658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:37.399408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3658
I20260927 00:41:37.399536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:37.399820 392 external_mini_cluster.cc:1428] Setting key 767c1ec023af215a64bf9eda91178078
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:37.410207 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.397144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34073 (local address 127.0.98.62:36825)
0927 00:41:37.397338 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:37.397343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:37.398268 (+ 925us) server_negotiation.cc:407] Connection header received
0927 00:41:37.399650 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:37.399653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:37.399708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:37.399786 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:37.401319 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.401884 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.403402 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.403884 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.404014 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.406479 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:37.406484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:37.406490 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:37.406532 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:37.408193 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.408680 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.408684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.408686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.408776 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.409184 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.409187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.409189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.409411 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:37.409540 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:37.409992 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 00:41:37.410090 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 00:41:37.411015 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.397342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36825 (local address 127.0.98.2:34073)
0927 00:41:37.398100 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:37.398161 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:37.399431 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:37.399955 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:37.399963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:37.400430 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:37.401142 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.401164 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.402034 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.402041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.403273 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.403282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.404024 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.404027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.404163 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.404758 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:37.404774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:37.406311 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:37.408302 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.408307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.408318 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.408564 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.408868 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.408870 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.408871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.409075 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.409528 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:37.409534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:37.409768 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 00:41:37.409916 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 00:41:37.410155 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:37.412261 3787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36825
I20260927 00:41:37.412492 3787 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:37.413021 3787 heartbeater.cc:511] Master 127.0.98.62:36825 requested a full tablet report, sending...
I20260927 00:41:37.414110 3462 ts_manager.cc:194] Registered new tserver with Master: d7715f7847f24718b72dab3fcd6b85a4 (127.0.98.2:40327)
I20260927 00:41:37.414709 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:37.427531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:37.534617 3792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:37.534878 3792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:37.534945 3792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:37.538893 3792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:37.538942 3792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:37.538996 3792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:37.543944 3792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:37.544824 3792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:37.545779 3792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:37.554795 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.554790 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:37.554795 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:37.555121 3792 server_base.cc:1034] running on GCE node
I20260927 00:41:37.555455 3792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:37.556031 3792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:37.557192 3792 hybrid_clock.cc:648] HybridClock initialized: now 1790469697557178 us; error 30 us; skew 500 ppm
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:37.559863 3792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:37.560750 3792 webserver.cc:460] Webserver started at http://127.0.98.3:43115/ using document root <none> and password file <none>
I20260927 00:41:37.561311 3792 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:37.561371 3792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:37.561527 3792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:37.563457 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "20496a84481543de9e9204f3b256dade"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "2279f11b38d39ff264f818470cd280ae"
server_key_iv: "9be92143b2bfa3973332b13a2afd0be9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.563952 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20496a84481543de9e9204f3b256dade"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "2279f11b38d39ff264f818470cd280ae"
server_key_iv: "9be92143b2bfa3973332b13a2afd0be9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.567682 3792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 00:41:37.570075 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.571269 3792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:37.571367 3792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "20496a84481543de9e9204f3b256dade"
format_stamp: "Formatted at 2026-09-27 00:41:37 on dist-test-slave-pkws"
server_key: "2279f11b38d39ff264f818470cd280ae"
server_key_iv: "9be92143b2bfa3973332b13a2afd0be9"
server_key_version: "encryptionkey@0"
I20260927 00:41:37.571445 3792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:37.622397 3792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:37.625720 3792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:37.625871 3792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:37.626399 3792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:37.627163 3792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:37.627218 3792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.627259 3792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:37.627279 3792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:37.636232 3792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43965
I20260927 00:41:37.636256 3920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43965 every 8 connection(s)
I20260927 00:41:37.637145 3792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:37.643671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3792
I20260927 00:41:37.643790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:37.644086 392 external_mini_cluster.cc:1428] Setting key 0853db3112f9b5d84ed2326d26f8aa84
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:37.650324 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.638879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42119 (local address 127.0.98.62:36825)
0927 00:41:37.638998 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:37.639002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:37.639753 (+ 751us) server_negotiation.cc:407] Connection header received
0927 00:41:37.641171 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:37.641174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:37.641242 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:37.641335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:37.642448 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.642990 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.644252 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.644502 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.644612 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.647175 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:37.647178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:37.647180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:37.647202 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:37.648418 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.648931 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.648934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.648936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.649024 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.649408 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.649410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.649411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.649572 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:37.649647 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:37.650091 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 00:41:37.650182 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:41:37.651144 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.639206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36825 (local address 127.0.98.3:42119)
0927 00:41:37.639612 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:37.639653 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:37.640965 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:37.641505 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:37.641513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:37.641852 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:37.642312 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.642324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.643119 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.643123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.644110 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:37.644135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.644649 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.644653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:37.644854 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.645493 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:37.645509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:37.647030 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:37.648525 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.648529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.648539 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.648773 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.649127 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:37.649131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:37.649136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:37.649288 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:37.649655 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:37.649660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:37.649867 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 00:41:37.650060 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 00:41:37.650344 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260927 00:41:37.652168 3921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36825
I20260927 00:41:37.652391 3921 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:37.652833 3921 heartbeater.cc:511] Master 127.0.98.62:36825 requested a full tablet report, sending...
I20260927 00:41:37.653774 3462 ts_manager.cc:194] Registered new tserver with Master: 20496a84481543de9e9204f3b256dade (127.0.98.3:43965)
I20260927 00:41:37.654325 3462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42119
I20260927 00:41:37.657616 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469697, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:37.681705 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:37.673923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.98.62:36825)
0927 00:41:37.674058 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:37.674063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:37.674252 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:41:37.674486 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:37.674489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:37.674531 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:37.674599 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:37.675252 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.675749 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.676528 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:37.676739 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:37.676818 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:37.679097 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:37.679100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:37.679102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:37.679124 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:37.680299 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.680659 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.680661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.680662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.680730 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:37.681017 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:37.681019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:37.681020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:37.681210 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:37.681316 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:37.681488 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:41:37.681594 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:41:37.682022 3462 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36556
I20260927 00:41:37.682796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3524
I20260927 00:41:37.688042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3658
I20260927 00:41:37.693302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3792
I20260927 00:41:37.698312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3430
2026-09-27T00:41:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2135 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3933](info): set up 2 sockets
Sep 27 00:41:37 dist-test-slave-pkws krb5kdc[3933](info): commencing operation
krb5kdc: starting...
W20260927 00:41:38.736826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:38 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:38.851994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:38.959565 3946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:38.959858 3946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:38.959939 3946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:38.963864 3946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:38.963945 3946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:38.963984 3946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:38.964036 3946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:38.964076 3946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:38.969125 3946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:38.970201 3946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:38.971155 3946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:38.979431 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:38.979458 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:38.979496 3946 server_base.cc:1034] running on GCE node
W20260927 00:41:38.979458 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:38.980233 3946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:38.981366 3946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:38.982545 3946 hybrid_clock.cc:648] HybridClock initialized: now 1790469698982524 us; error 36 us; skew 500 ppm
Sep 27 00:41:38 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:38.985188 3946 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:38.986012 3946 webserver.cc:460] Webserver started at http://127.0.98.62:39623/ using document root <none> and password file <none>
I20260927 00:41:38.986544 3946 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:38.986652 3946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:38.986881 3946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:38.988961 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "0989f915b27545849f40f33831b1fea4"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-pkws"
server_key: "c056ea469154b7d36317ff76da660aca"
server_key_iv: "03e7fd0403d16ad24b1864ae1f81f52e"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.989521 3946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "0989f915b27545849f40f33831b1fea4"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-pkws"
server_key: "c056ea469154b7d36317ff76da660aca"
server_key_iv: "03e7fd0403d16ad24b1864ae1f81f52e"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.993276 3946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:38.995605 3961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:38.996811 3946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:38.996960 3946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "0989f915b27545849f40f33831b1fea4"
format_stamp: "Formatted at 2026-09-27 00:41:38 on dist-test-slave-pkws"
server_key: "c056ea469154b7d36317ff76da660aca"
server_key_iv: "03e7fd0403d16ad24b1864ae1f81f52e"
server_key_version: "encryptionkey@0"
I20260927 00:41:38.997068 3946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.016522 3946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.019470 3946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.019665 3946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.027153 3946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35797
I20260927 00:41:39.027235 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35797 every 8 connection(s)
I20260927 00:41:39.028069 3946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:39.030673 4014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:39.035912 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: Bootstrap starting.
I20260927 00:41:39.037894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 3946
I20260927 00:41:39.038017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:39.038303 392 external_mini_cluster.cc:1428] Setting key ea7cc06cbb7e9df9493dd55cf04c20e0
I20260927 00:41:39.038683 4014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:39.039530 4014 log.cc:824] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:39.041882 4014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: No bootstrap required, opened a new log
I20260927 00:41:39.045289 4014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260927 00:41:39.045521 4014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:39.045604 4014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0989f915b27545849f40f33831b1fea4, State: Initialized, Role: FOLLOWER
I20260927 00:41:39.046012 4014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260927 00:41:39.046164 4014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:39.046243 4014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:39.046353 4014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.047508 4014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260927 00:41:39.047873 4014 leader_election.cc:302] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0989f915b27545849f40f33831b1fea4; no voters:
I20260927 00:41:39.048233 4014 leader_election.cc:288] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:39.048364 4019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:39.048623 4019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [term 1 LEADER]: Becoming Leader. State: Replica: 0989f915b27545849f40f33831b1fea4, State: Running, Role: LEADER
I20260927 00:41:39.049041 4019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } }
I20260927 00:41:39.049574 4014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:39.050112 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0989f915b27545849f40f33831b1fea4. Latest consensus state: current_term: 1 leader_uuid: "0989f915b27545849f40f33831b1fea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } } }
I20260927 00:41:39.050225 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:39.050616 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0989f915b27545849f40f33831b1fea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0989f915b27545849f40f33831b1fea4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35797 } } }
I20260927 00:41:39.050719 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:39.054093 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.039452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.0.98.62:35797)
0927 00:41:39.039916 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.039936 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.039980 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:41:39.040697 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.040715 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.041138 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.041462 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.042357 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.043332 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.044184 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.044650 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.044815 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.047780 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.047787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.047802 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.047830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.049998 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.051302 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.051307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.051313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.051453 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.051781 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.051785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.051787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.052328 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.052551 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.052835 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.053157 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
W20260927 00:41:39.054689 4035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:39.054774 4035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:39.054888 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:39.055557 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:39.060351 4024 catalog_manager.cc:1347] Generated new cluster ID: 1f799f31878841f299f1dbde0db180b0
I20260927 00:41:39.060413 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:39.081631 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:39.082630 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:39.090202 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0989f915b27545849f40f33831b1fea4: Generated new TSK 0
I20260927 00:41:39.090883 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:39.117905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.224844 4040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.225157 4040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.225226 4040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.229208 4040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.229278 4040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.229364 4040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:39.234325 4040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:39.235273 4040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.236220 4040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:39.245211 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.245229 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.245324 4040 server_base.cc:1034] running on GCE node
W20260927 00:41:39.245229 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.245820 4040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.246387 4040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.247568 4040 hybrid_clock.cc:648] HybridClock initialized: now 1790469699247546 us; error 34 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.250216 4040 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:39.251072 4040 webserver.cc:460] Webserver started at http://127.0.98.1:36627/ using document root <none> and password file <none>
I20260927 00:41:39.251600 4040 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.251685 4040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.251892 4040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:39.253820 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "00e8b27456a5418eb8ffeecfb2fb1828"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "491070432f4e4a88b136ea7f91cb9071"
server_key_iv: "9d54342f06f163fe0354b334900371c9"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.254354 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00e8b27456a5418eb8ffeecfb2fb1828"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "491070432f4e4a88b136ea7f91cb9071"
server_key_iv: "9d54342f06f163fe0354b334900371c9"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.257722 4040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:39.259873 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.261015 4040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 00:41:39.261137 4040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "00e8b27456a5418eb8ffeecfb2fb1828"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "491070432f4e4a88b136ea7f91cb9071"
server_key_iv: "9d54342f06f163fe0354b334900371c9"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.261227 4040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.272164 4040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.274873 4040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.275056 4040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.275601 4040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:39.276360 4040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:39.276412 4040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.276476 4040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:39.276506 4040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.284564 4040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39679
I20260927 00:41:39.284646 4168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39679 every 8 connection(s)
I20260927 00:41:39.285440 4040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:39.293685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4040
I20260927 00:41:39.293815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:39.294124 392 external_mini_cluster.cc:1428] Setting key 633a5a69056460a29b1cc055bbe1ba5b
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.298753 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.287108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54497 (local address 127.0.98.62:35797)
0927 00:41:39.287256 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.287261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.288045 (+ 784us) server_negotiation.cc:407] Connection header received
0927 00:41:39.289342 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.289344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.289391 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.289465 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.290530 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.291057 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.292143 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.292378 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.292458 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.294976 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.294979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.294982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.295004 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.296421 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.297058 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.297064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.297068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.297167 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.297496 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.297501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.297503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.297742 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.297845 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.298368 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.298507 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:41:39.299449 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.287478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.1:54497)
0927 00:41:39.287908 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.287947 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.289156 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.289618 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.289624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.289962 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:39.290410 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.290423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.291183 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.291186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.292023 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.292031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.292504 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.292507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.292639 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.293252 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:39.293268 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:39.294815 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:39.296534 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.296543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.296561 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.296949 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.297264 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.297266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.297267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.297399 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.297868 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:39.297874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:39.298101 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.298286 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.298529 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260927 00:41:39.300613 4169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260927 00:41:39.300872 4169 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.301426 4169 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260927 00:41:39.302843 3978 ts_manager.cc:194] Registered new tserver with Master: 00e8b27456a5418eb8ffeecfb2fb1828 (127.0.98.1:39679)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:39.303999 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:39.320022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.427475 4174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.427729 4174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.427772 4174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.431762 4174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.431823 4174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.431922 4174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:39.437011 4174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:39.437968 4174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.438939 4174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:39.448088 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.448117 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.448241 4174 server_base.cc:1034] running on GCE node
W20260927 00:41:39.448297 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.448697 4174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.449365 4174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.450537 4174 hybrid_clock.cc:648] HybridClock initialized: now 1790469699450519 us; error 30 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.453150 4174 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:39.453939 4174 webserver.cc:460] Webserver started at http://127.0.98.2:37333/ using document root <none> and password file <none>
I20260927 00:41:39.454497 4174 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.454582 4174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.454798 4174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:39.456727 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "58bc35c35c0e442399e29b309a3e2605"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "f448e7529c7571e373d0405de4f3245e"
server_key_iv: "10c65d3a68870923ec4971b51effe50b"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.457332 4174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58bc35c35c0e442399e29b309a3e2605"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "f448e7529c7571e373d0405de4f3245e"
server_key_iv: "10c65d3a68870923ec4971b51effe50b"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.460760 4174 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:39.463042 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.464205 4174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:39.464334 4174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "58bc35c35c0e442399e29b309a3e2605"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "f448e7529c7571e373d0405de4f3245e"
server_key_iv: "10c65d3a68870923ec4971b51effe50b"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.464432 4174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.475927 4174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.478590 4174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.478772 4174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.479288 4174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:39.480060 4174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:39.480134 4174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.480196 4174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:39.480243 4174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.488824 4174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34595
I20260927 00:41:39.488869 4302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34595 every 8 connection(s)
I20260927 00:41:39.489753 4174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:39.495278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4174
I20260927 00:41:39.495397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:39.495672 392 external_mini_cluster.cc:1428] Setting key de62cd78b65f5bc959fa6a77ced90e74
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.504204 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47705 (local address 127.0.98.62:35797)
0927 00:41:39.491524 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.491528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.492389 (+ 861us) server_negotiation.cc:407] Connection header received
0927 00:41:39.493724 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.493727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.493772 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.493845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.494941 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.495632 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.496959 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497212 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497294 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.500430 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.500438 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.500441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.500462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.501985 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.502638 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.502640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.502642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.502711 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.503132 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.503134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.503135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.503305 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.503383 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.503996 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.504086 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:41:39.505424 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.491800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.2:47705)
0927 00:41:39.492242 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.492282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.493535 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.493998 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.494005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.494374 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:39.494822 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.494834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.495754 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.495758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.496725 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.496735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.497359 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.497363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.497577 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.498156 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:39.498181 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:39.500243 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:39.502107 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.502116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.502134 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.502531 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.502798 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.502803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.502808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.503017 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.503409 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:39.503414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:39.503749 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.503956 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.504319 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:39.506560 4303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260927 00:41:39.506834 4303 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.507381 4303 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260927 00:41:39.508442 3978 ts_manager.cc:194] Registered new tserver with Master: 58bc35c35c0e442399e29b309a3e2605 (127.0.98.2:34595)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:39.509093 3978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:39.524087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:39.630589 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:39.630831 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:39.630910 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:39.634853 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:39.634898 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:39.634984 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:39.639951 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:39.640899 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:39.641870 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:39.651091 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:39.651113 4314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.651130 4308 server_base.cc:1034] running on GCE node
W20260927 00:41:39.651096 4316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:39.651679 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:39.652266 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:39.653414 4308 hybrid_clock.cc:648] HybridClock initialized: now 1790469699653402 us; error 29 us; skew 500 ppm
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:39.656028 4308 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:39.656864 4308 webserver.cc:460] Webserver started at http://127.0.98.3:34313/ using document root <none> and password file <none>
I20260927 00:41:39.657433 4308 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:39.657493 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:39.657721 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:39.659669 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f9b4cabd8b2499cb4a1784f40fb7edd"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "8054dfa80ea6e057a4a5c17aa37010b8"
server_key_iv: "62f49a3edeea41c9ba03e6393d551cea"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.660220 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f9b4cabd8b2499cb4a1784f40fb7edd"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "8054dfa80ea6e057a4a5c17aa37010b8"
server_key_iv: "62f49a3edeea41c9ba03e6393d551cea"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.663753 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:39.666213 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.667306 4308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:39.667430 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "1f9b4cabd8b2499cb4a1784f40fb7edd"
format_stamp: "Formatted at 2026-09-27 00:41:39 on dist-test-slave-pkws"
server_key: "8054dfa80ea6e057a4a5c17aa37010b8"
server_key_iv: "62f49a3edeea41c9ba03e6393d551cea"
server_key_version: "encryptionkey@0"
I20260927 00:41:39.667527 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:39.712469 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:39.715854 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:39.716048 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:39.716630 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:39.717483 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:39.717528 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.717592 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:39.717623 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:39.725927 4308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44185
I20260927 00:41:39.725955 4436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44185 every 8 connection(s)
I20260927 00:41:39.726784 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:39.729653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4308
I20260927 00:41:39.729786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:39.730055 392 external_mini_cluster.cc:1428] Setting key aa7ef582248cca7d8e8feb50895a3a92
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.739833 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.728464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41131 (local address 127.0.98.62:35797)
0927 00:41:39.728598 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.728602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.729457 (+ 855us) server_negotiation.cc:407] Connection header received
0927 00:41:39.730927 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.730930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.730983 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.731063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.732253 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.732775 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.733893 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.734110 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.734223 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.736545 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.736548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.736551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.736581 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.737901 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.738413 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.738417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.738419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.738499 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.738830 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.738833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.738835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.739007 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.739106 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.739571 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.739684 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:41:39.740605 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.728829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35797 (local address 127.0.98.3:41131)
0927 00:41:39.729311 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:39.729352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:39.730725 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:39.731234 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:39.731241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:39.731573 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:39.732115 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.732131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.732922 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.732925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.733771 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:39.733779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.734255 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.734259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:39.734420 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.734984 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:39.734999 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:39.736386 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:39.738012 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.738018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.738032 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.738309 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.738587 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:39.738592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:39.738594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:39.738731 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:39.739116 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:39.739124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:39.739352 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 00:41:39.739533 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:41:39.739839 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":109,"threads_started":1}
I20260927 00:41:39.741637 4437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35797
I20260927 00:41:39.741837 4437 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:39.742270 4437 heartbeater.cc:511] Master 127.0.98.62:35797 requested a full tablet report, sending...
I20260927 00:41:39.743309 3977 ts_manager.cc:194] Registered new tserver with Master: 1f9b4cabd8b2499cb4a1784f40fb7edd (127.0.98.3:44185)
I20260927 00:41:39.743875 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41131
I20260927 00:41:39.754796 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:39.780184 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.770768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.0.98.1:39679)
0927 00:41:39.771186 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.771207 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.771236 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:41:39.771384 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.771390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.771590 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.771740 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.772424 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.773247 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.774076 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.774447 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.774570 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.776972 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.776980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.776993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.777040 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.778569 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.778976 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.778981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.778982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.779057 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.779320 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.779322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.779324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.779543 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.779708 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.779881 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.779996 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[3933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:39.790596 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:39.783160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.98.62:35797)
0927 00:41:39.783316 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:39.783320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:39.783371 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:41:39.783471 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:39.783473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:39.783510 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:39.783576 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:39.784131 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.784614 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.785447 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:39.785650 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:39.785722 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:39.787914 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:39.787917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:39.787919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:39.787940 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:39.789159 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.789509 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.789511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.789512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.789579 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:39.789794 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:39.789796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:39.789797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:39.789988 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:39.790086 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:39.790315 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:41:39.790400 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 00:41:39.790957 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56780
I20260927 00:41:39.791545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4040
I20260927 00:41:39.797124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4174
I20260927 00:41:39.802321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4308
I20260927 00:41:39.808187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 3946
2026-09-27T00:41:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2111 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[4449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[4449](info): set up 2 sockets
Sep 27 00:41:39 dist-test-slave-pkws krb5kdc[4449](info): commencing operation
krb5kdc: starting...
W20260927 00:41:40.853000 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:40 dist-test-slave-pkws krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:40.971673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:41.077546 4462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.077819 4462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.077899 4462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.082729 4462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:41.082815 4462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.082854 4462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.082906 4462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:41.082952 4462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:41.088485 4462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:41.089583 4462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.090585 4462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.098666 4467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.098760 4470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.098832 4462 server_base.cc:1034] running on GCE node
W20260927 00:41:41.098721 4468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.099390 4462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.100457 4462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.101639 4462 hybrid_clock.cc:648] HybridClock initialized: now 1790469701101623 us; error 30 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.110471 4462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:41.111183 4462 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.111256 4462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.111420 4462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:41.113462 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "dc4e845d9bda47cc95fbc399b2f98559"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "59744ad0ad45d389b56f3309fda714e1"
server_key_iv: "6f68a86dbf0e938e712fd31af396fe10"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.113976 4462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc4e845d9bda47cc95fbc399b2f98559"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "59744ad0ad45d389b56f3309fda714e1"
server_key_iv: "6f68a86dbf0e938e712fd31af396fe10"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.117668 4462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:41.119983 4476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.121284 4462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:41.121440 4462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "dc4e845d9bda47cc95fbc399b2f98559"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "59744ad0ad45d389b56f3309fda714e1"
server_key_iv: "6f68a86dbf0e938e712fd31af396fe10"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.121537 4462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.141659 4462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.144390 4462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.144586 4462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.151124 4462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39601
I20260927 00:41:41.151382 4528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39601 every 8 connection(s)
I20260927 00:41:41.151763 4462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:41.154356 4529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:41.157436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4462
I20260927 00:41:41.157557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:41.157842 392 external_mini_cluster.cc:1428] Setting key 735e60fa876ff9a39f451923d78d3ecb
I20260927 00:41:41.160075 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: Bootstrap starting.
I20260927 00:41:41.162546 4529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:41.163383 4529 log.cc:824] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.165426 4529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: No bootstrap required, opened a new log
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:41.168836 4529 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260927 00:41:41.169123 4529 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:41.169196 4529 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4e845d9bda47cc95fbc399b2f98559, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.169642 4529 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260927 00:41:41.169783 4529 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:41.169866 4529 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:41.169979 4529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:41.171016 4529 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260927 00:41:41.171344 4529 leader_election.cc:302] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc4e845d9bda47cc95fbc399b2f98559; no voters:
I20260927 00:41:41.171633 4529 leader_election.cc:288] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:41.171736 4534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:41.171954 4534 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [term 1 LEADER]: Becoming Leader. State: Replica: dc4e845d9bda47cc95fbc399b2f98559, State: Running, Role: LEADER
I20260927 00:41:41.171980 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.159090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.0.98.62:39601)
0927 00:41:41.159502 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.159524 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.159558 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:41:41.160165 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.160188 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.160545 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.160815 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:41.161742 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.162809 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.163919 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.164466 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.164583 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.167251 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:41.167256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:41.167267 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:41.167292 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:41.168959 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.169461 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.169467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.169476 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.169575 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.169856 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.169859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.169861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.170344 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:41.170574 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.170765 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.171055 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":148,"threads_started":1}
I20260927 00:41:41.172325 4534 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260927 00:41:41.172796 4529 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:41.173501 4534 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4e845d9bda47cc95fbc399b2f98559. Latest consensus state: current_term: 1 leader_uuid: "dc4e845d9bda47cc95fbc399b2f98559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260927 00:41:41.173707 4534 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:41.173936 4536 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc4e845d9bda47cc95fbc399b2f98559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc4e845d9bda47cc95fbc399b2f98559" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260927 00:41:41.174077 4536 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:41.174541 4543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:41.177614 4543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:41.182895 4543 catalog_manager.cc:1347] Generated new cluster ID: 888150c39c8f44a080bb6b30f1f00a17
I20260927 00:41:41.182963 4543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:41.196233 4543 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:41.197209 4543 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:41.201408 4543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: Generated new TSK 0
I20260927 00:41:41.201974 4543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:41.235854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:41.359663 4555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.359920 4555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.360002 4555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.364027 4555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.364077 4555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.364133 4555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:41.369194 4555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:41.370128 4555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.371124 4555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.380275 4560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.380334 4555 server_base.cc:1034] running on GCE node
W20260927 00:41:41.380291 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.380297 4563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.380959 4555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.381623 4555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.382798 4555 hybrid_clock.cc:648] HybridClock initialized: now 1790469701382786 us; error 30 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.385535 4555 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:41.386252 4555 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.386337 4555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.386549 4555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:41.388445 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "22329153174443a387d8639d363c15ae"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "be1600f1299698d2a19839d051d2f595"
server_key_iv: "9c195139649c3c7ba386eb6c0ff54bbb"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.389066 4555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22329153174443a387d8639d363c15ae"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "be1600f1299698d2a19839d051d2f595"
server_key_iv: "9c195139649c3c7ba386eb6c0ff54bbb"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.392784 4555 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:41.395138 4569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.396512 4555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:41.396633 4555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "22329153174443a387d8639d363c15ae"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "be1600f1299698d2a19839d051d2f595"
server_key_iv: "9c195139649c3c7ba386eb6c0ff54bbb"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.396737 4555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.410483 4555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.414378 4555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.414575 4555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.415134 4555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.415936 4555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:41.416004 4555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.416069 4555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:41.416101 4555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.424995 4555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41239
I20260927 00:41:41.425030 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41239 every 8 connection(s)
I20260927 00:41:41.425652 4555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:41.432178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4555
I20260927 00:41:41.432302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:41.432610 392 external_mini_cluster.cc:1428] Setting key 943c2adb03bcb2f88bb213fa7bf8dfbf
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:41.440107 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.427350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55351 (local address 127.0.98.62:39601)
0927 00:41:41.427495 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.427499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.428380 (+ 881us) server_negotiation.cc:407] Connection header received
0927 00:41:41.429817 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.429819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.429866 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.429943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:41.431088 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.431601 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.433310 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.433607 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.433711 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.436602 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:41.436608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:41.436612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:41.436645 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:41.438171 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.438671 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.438676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.438677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.438763 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.439092 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.439095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.439097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.439272 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:41.439374 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.439837 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.439948 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:41.440868 4685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.427759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.1:55351)
0927 00:41:41.428232 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.428275 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.429607 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.430086 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.430092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.430491 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:41.430967 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.430981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.431716 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.431719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.433103 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.433112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.433767 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.433771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.433973 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.434579 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:41.434603 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:41.436445 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:41.438294 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.438300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.438314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.438566 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.438852 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.438856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.438858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.438993 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.439384 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:41.439392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:41.439618 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.439798 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.440092 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:41.442226 4683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260927 00:41:41.442442 4683 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:41.442873 4683 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260927 00:41:41.444355 4493 ts_manager.cc:194] Registered new tserver with Master: 22329153174443a387d8639d363c15ae (127.0.98.1:41239)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:41.445583 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:41.459307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:41.563966 4688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.564231 4688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.564308 4688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.568334 4688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.568420 4688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.568619 4688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:41.574257 4688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:41.575322 4688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.576356 4688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.585152 4696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.585130 4694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.585208 4693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.585384 4688 server_base.cc:1034] running on GCE node
I20260927 00:41:41.585731 4688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.586315 4688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.587487 4688 hybrid_clock.cc:648] HybridClock initialized: now 1790469701587475 us; error 28 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.590163 4688 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:41.590845 4688 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.590931 4688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.591138 4688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:41.593146 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8cf883cf976400e9299fccba8e0134c"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "bec7079582121ee16757ec77e871b131"
server_key_iv: "a07f023169ece6885a6835454923a653"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.593680 4688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8cf883cf976400e9299fccba8e0134c"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "bec7079582121ee16757ec77e871b131"
server_key_iv: "a07f023169ece6885a6835454923a653"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.597178 4688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:41.599704 4702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.600983 4688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:41.601109 4688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "a8cf883cf976400e9299fccba8e0134c"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "bec7079582121ee16757ec77e871b131"
server_key_iv: "a07f023169ece6885a6835454923a653"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.601258 4688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.617606 4688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.620325 4688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.620500 4688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.621060 4688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.621848 4688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:41.621913 4688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.621977 4688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:41.622025 4688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.630777 4688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43213
I20260927 00:41:41.630797 4815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43213 every 8 connection(s)
I20260927 00:41:41.631323 4688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:41.634845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4688
I20260927 00:41:41.634981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:41.635337 392 external_mini_cluster.cc:1428] Setting key 94ed2dbfa83834cb4d7dc65dc25b9b1b
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:41.644790 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.632983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47793 (local address 127.0.98.62:39601)
0927 00:41:41.633146 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.633150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.633927 (+ 777us) server_negotiation.cc:407] Connection header received
0927 00:41:41.635281 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.635284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.635338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.635442 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:41.636741 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.637338 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.638461 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.638691 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.638797 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.641144 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:41.641151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:41.641158 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:41.641200 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:41.642809 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.643268 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.643272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.643274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.643366 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.643662 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.643666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.643668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.643918 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:41.644032 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.644460 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.644577 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:41:41.645458 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.633379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.2:47793)
0927 00:41:41.633793 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.633831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.635035 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.635609 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.635615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.636035 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:41.636592 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.636609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.637456 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.637459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.638340 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.638348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.638831 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.638835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.638989 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.639547 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:41.639563 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:41.640963 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:41.642920 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.642924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.642934 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.643167 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.643455 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.643457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.643459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.643575 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.644021 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:41.644026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:41.644224 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.644384 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.644618 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260927 00:41:41.646574 4816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260927 00:41:41.646802 4816 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:41.647233 4816 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:41.648360 4493 ts_manager.cc:194] Registered new tserver with Master: a8cf883cf976400e9299fccba8e0134c (127.0.98.2:43213)
I20260927 00:41:41.648957 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:41.663851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:41.769573 4821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:41.769804 4821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:41.769846 4821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:41.774016 4821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:41.774075 4821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:41.774132 4821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:41.779413 4821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:41.780326 4821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:41.781299 4821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:41.790309 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.790337 4827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:41.790331 4829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:41.790385 4821 server_base.cc:1034] running on GCE node
I20260927 00:41:41.790943 4821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:41.791507 4821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:41.792680 4821 hybrid_clock.cc:648] HybridClock initialized: now 1790469701792646 us; error 44 us; skew 500 ppm
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:41.795367 4821 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:41.796121 4821 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:41.796209 4821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:41.796422 4821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:41.798408 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf7e125286ba444f8ea4e522ec5a3f1e"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "b0fe0aae2d2d92f8f44df380323f2dac"
server_key_iv: "72e7ed292dfead3ec33cea5e03be015f"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.798969 4821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf7e125286ba444f8ea4e522ec5a3f1e"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "b0fe0aae2d2d92f8f44df380323f2dac"
server_key_iv: "72e7ed292dfead3ec33cea5e03be015f"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.802570 4821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:41.804898 4835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.806051 4821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:41.806175 4821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "bf7e125286ba444f8ea4e522ec5a3f1e"
format_stamp: "Formatted at 2026-09-27 00:41:41 on dist-test-slave-pkws"
server_key: "b0fe0aae2d2d92f8f44df380323f2dac"
server_key_iv: "72e7ed292dfead3ec33cea5e03be015f"
server_key_version: "encryptionkey@0"
I20260927 00:41:41.806260 4821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:41.837414 4821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:41.840127 4821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:41.840317 4821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:41.840845 4821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:41.841704 4821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:41.841769 4821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.841835 4821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:41.841881 4821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:41.850255 4821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33823
I20260927 00:41:41.850266 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33823 every 8 connection(s)
I20260927 00:41:41.850790 4821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:41.859505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4821
I20260927 00:41:41.859623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:41.859969 392 external_mini_cluster.cc:1428] Setting key 9ad420840707b8d2de67d9aa18150786
Sep 27 00:41:41 dist-test-slave-pkws krb5kdc[4449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:41.865818 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.852406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37153 (local address 127.0.98.62:39601)
0927 00:41:41.852527 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.852530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.853281 (+ 751us) server_negotiation.cc:407] Connection header received
0927 00:41:41.854625 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.854628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.854684 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.854788 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:41.856046 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.856845 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.858687 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.859088 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.859192 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.862184 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:41.862188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:41.862191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:41.862218 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:41.863927 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.864472 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.864475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.864476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.864550 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.864800 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.864802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.864804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.865025 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:41.865102 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.865574 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.865671 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:41:41.866604 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.852658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.3:37153)
0927 00:41:41.853119 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.853159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.854419 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.854954 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.854961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.855296 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:41.855857 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.855880 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.857028 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.857031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.858541 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.858562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.859394 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.859400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.859625 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.860293 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:41.860312 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:41.862005 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:41.864038 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.864043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.864058 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.864385 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.864613 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:41.864615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:41.864616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:41.864728 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:41.865110 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:41.865114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:41.865377 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.865532 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.865769 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260927 00:41:41.867604 4949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260927 00:41:41.867854 4949 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:41.868285 4949 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260927 00:41:41.869270 4493 ts_manager.cc:194] Registered new tserver with Master: bf7e125286ba444f8ea4e522ec5a3f1e (127.0.98.3:33823)
I20260927 00:41:41.869778 4493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37153
I20260927 00:41:41.874099 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:41.883059 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.875960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.0.98.62:39601)
0927 00:41:41.876114 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.876119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.876333 (+ 214us) server_negotiation.cc:407] Connection header received
0927 00:41:41.876462 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.876466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.876511 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.876589 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:41.877322 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.877831 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.878635 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.878851 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.878972 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.880187 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:41.880191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:41.880193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:41.880217 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:41.881406 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.881819 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.881822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.881824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.881898 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:41.882182 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:41.882187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:41.882189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:41.882402 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:41.882548 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.882768 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.882894 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:41:41.885607 4493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:41.888456 4493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:41.903086 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.896763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33823 (local address 127.0.0.1:49788)
0927 00:41:41.897303 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.897355 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.897488 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.898482 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.898490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.898519 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:41.898675 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.898685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.900130 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.900134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.901563 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.901572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.902556 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.902559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.902657 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.902690 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.902797 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.902904 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":182,"thread_start_us":70,"threads_started":1}
I20260927 00:41:41.903087 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.897591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.0.98.3:33823)
0927 00:41:41.897939 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.897961 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.897985 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:41:41.898068 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.898075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.898275 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.898453 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:41.898803 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.900016 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.901681 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.902428 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.902545 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.902677 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.902837 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.902929 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":144,"threads_started":1}
I20260927 00:41:41.903321 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.897923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.98.2:43213)
0927 00:41:41.898311 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.898325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.898585 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:41:41.898797 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.898805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.898986 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.899133 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:41.899293 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.900150 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.901986 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.902550 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.902625 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.902751 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.903052 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.903151 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":87,"threads_started":1}
I20260927 00:41:41.903828 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.897509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43213 (local address 127.0.0.1:38130)
0927 00:41:41.898482 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.898494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.898651 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.899106 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.899109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.899125 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:41.899204 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.899210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.900271 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.900274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.901843 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.901854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.902835 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.902838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.902912 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.902930 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.903607 (+ 677us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.903692 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":130,"thread_start_us":75,"threads_started":1}
I20260927 00:41:41.903873 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.897023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.0.98.1:41239)
0927 00:41:41.897751 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:41.897778 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:41.898089 (+ 311us) server_negotiation.cc:407] Connection header received
0927 00:41:41.898157 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:41.898161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:41.898301 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:41.898451 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:41.898868 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.899998 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.901966 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.902727 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.902853 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.902970 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:41.903344 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 00:41:41.903450 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":83,"threads_started":1}
I20260927 00:41:41.904280 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:41.896783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41239 (local address 127.0.0.1:39074)
0927 00:41:41.897893 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:41.897906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:41.898000 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:41.898618 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:41.898620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:41.898637 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:41.898747 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.898754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.900181 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.900183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.901843 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:41.901854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:41.903032 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:41.903035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:41.903127 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:41.903151 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:41:41.903302 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 00:41:41.903374 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":189,"spinlock_wait_cycles":2944,"thread_start_us":134,"threads_started":1}
I20260927 00:41:41.906174 4883 tablet_service.cc:1467] Processing CreateTablet for tablet 529f8cb821654391bff661d9a0f3626d (DEFAULT_TABLE table=test-table [id=b681288b2d1c41c6aa1cc3c181a97064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:41.907104 4883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f8cb821654391bff661d9a0f3626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:41.906754 4750 tablet_service.cc:1467] Processing CreateTablet for tablet 529f8cb821654391bff661d9a0f3626d (DEFAULT_TABLE table=test-table [id=b681288b2d1c41c6aa1cc3c181a97064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:41.906873 4617 tablet_service.cc:1467] Processing CreateTablet for tablet 529f8cb821654391bff661d9a0f3626d (DEFAULT_TABLE table=test-table [id=b681288b2d1c41c6aa1cc3c181a97064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:41.907778 4617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f8cb821654391bff661d9a0f3626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:41.907778 4750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529f8cb821654391bff661d9a0f3626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:41.913499 4966 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Bootstrap starting.
I20260927 00:41:41.915055 4967 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Bootstrap starting.
I20260927 00:41:41.915712 4966 tablet_bootstrap.cc:654] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:41.916719 4967 tablet_bootstrap.cc:654] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:41.916720 4966 log.cc:824] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.917243 4968 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Bootstrap starting.
I20260927 00:41:41.917953 4967 log.cc:824] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.918946 4966 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: No bootstrap required, opened a new log
I20260927 00:41:41.919126 4966 ts_tablet_manager.cc:1397] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:41.919137 4968 tablet_bootstrap.cc:654] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:41.919827 4967 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: No bootstrap required, opened a new log
I20260927 00:41:41.920006 4967 ts_tablet_manager.cc:1397] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:41:41.920145 4968 log.cc:824] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:41.921666 4968 tablet_bootstrap.cc:492] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: No bootstrap required, opened a new log
I20260927 00:41:41.921847 4968 ts_tablet_manager.cc:1397] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:41:41.922963 4966 raft_consensus.cc:348] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.923246 4966 raft_consensus.cc:374] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:41.923305 4966 raft_consensus.cc:729] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7e125286ba444f8ea4e522ec5a3f1e, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.923733 4966 consensus_queue.cc:260] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.923679 4967 raft_consensus.cc:348] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.923923 4967 raft_consensus.cc:374] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:41.923979 4967 raft_consensus.cc:729] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22329153174443a387d8639d363c15ae, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.924401 4949 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260927 00:41:41.924875 4967 consensus_queue.cc:260] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.924721 4966 ts_tablet_manager.cc:1428] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:41.924916 4968 raft_consensus.cc:348] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.925259 4968 raft_consensus.cc:374] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:41.925309 4968 raft_consensus.cc:729] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf883cf976400e9299fccba8e0134c, State: Initialized, Role: FOLLOWER
I20260927 00:41:41.925838 4967 ts_tablet_manager.cc:1428] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 00:41:41.925876 4968 consensus_queue.cc:260] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:41.926139 4683 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260927 00:41:41.926524 4816 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260927 00:41:41.926769 4968 ts_tablet_manager.cc:1428] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 00:41:41.927322 4684 tablet.cc:2369] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:41:42.101605 4950 tablet.cc:2369] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:41:42.132368 4817 tablet.cc:2369] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:42.278501 4973 raft_consensus.cc:482] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:42.278748 4973 raft_consensus.cc:504] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:42.279882 4973 leader_election.cc:288] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf7e125286ba444f8ea4e522ec5a3f1e (127.0.98.3:33823), a8cf883cf976400e9299fccba8e0134c (127.0.98.2:43213)
I20260927 00:41:42.284152 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.280313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56699 (local address 127.0.98.3:33823)
0927 00:41:42.280464 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.280470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.280617 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:41:42.280762 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.280768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.280824 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.280932 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:42.281268 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.281848 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283083 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.283553 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283633 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.283682 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.283928 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.284013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:41:42.284152 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.280193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33823 (local address 127.0.98.1:56699)
0927 00:41:42.280491 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.280507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.280649 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.280944 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.280948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.280975 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:42.281114 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.281120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.281972 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.281976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.282955 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.282964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.283670 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.283798 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.283825 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.283901 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.283968 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260927 00:41:42.284409 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.280754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50939 (local address 127.0.98.2:43213)
0927 00:41:42.280955 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.280959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.280973 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:42.281097 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.281100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.281151 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.281259 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:42.281592 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.282211 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283330 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.283808 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283919 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.283973 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.284174 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.284258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 00:41:42.284489 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.280592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43213 (local address 127.0.98.1:50939)
0927 00:41:42.280877 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.280912 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.280996 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.281268 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.281271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.281287 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:42.281398 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.281404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.282337 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.282339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.283174 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:42.283184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.283940 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.283943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:42.284040 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.284062 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.284138 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.284347 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"spinlock_wait_cycles":768,"thread_start_us":105,"threads_started":1}
I20260927 00:41:42.285069 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f8cb821654391bff661d9a0f3626d" candidate_uuid: "22329153174443a387d8639d363c15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8cf883cf976400e9299fccba8e0134c" is_pre_election: true
I20260927 00:41:42.285108 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f8cb821654391bff661d9a0f3626d" candidate_uuid: "22329153174443a387d8639d363c15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" is_pre_election: true
I20260927 00:41:42.285389 4903 raft_consensus.cc:2454] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22329153174443a387d8639d363c15ae in term 0.
I20260927 00:41:42.285382 4770 raft_consensus.cc:2454] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22329153174443a387d8639d363c15ae in term 0.
I20260927 00:41:42.285884 4573 leader_election.cc:302] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22329153174443a387d8639d363c15ae, bf7e125286ba444f8ea4e522ec5a3f1e; no voters:
I20260927 00:41:42.286134 4973 raft_consensus.cc:2781] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:42.286219 4973 raft_consensus.cc:482] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:42.286262 4973 raft_consensus.cc:3037] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:42.287277 4973 raft_consensus.cc:504] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:42.287631 4973 leader_election.cc:288] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [CANDIDATE]: Term 1 election: Requested vote from peers bf7e125286ba444f8ea4e522ec5a3f1e (127.0.98.3:33823), a8cf883cf976400e9299fccba8e0134c (127.0.98.2:43213)
I20260927 00:41:42.287982 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f8cb821654391bff661d9a0f3626d" candidate_uuid: "22329153174443a387d8639d363c15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e"
I20260927 00:41:42.288085 4903 raft_consensus.cc:3037] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:42.289050 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529f8cb821654391bff661d9a0f3626d" candidate_uuid: "22329153174443a387d8639d363c15ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8cf883cf976400e9299fccba8e0134c"
I20260927 00:41:42.289184 4770 raft_consensus.cc:3037] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:42.289229 4903 raft_consensus.cc:2454] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22329153174443a387d8639d363c15ae in term 1.
I20260927 00:41:42.289525 4573 leader_election.cc:302] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22329153174443a387d8639d363c15ae, bf7e125286ba444f8ea4e522ec5a3f1e; no voters:
I20260927 00:41:42.289685 4973 raft_consensus.cc:2781] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:42.289918 4973 raft_consensus.cc:686] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 1 LEADER]: Becoming Leader. State: Replica: 22329153174443a387d8639d363c15ae, State: Running, Role: LEADER
I20260927 00:41:42.290246 4973 consensus_queue.cc:237] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } }
I20260927 00:41:42.290418 4770 raft_consensus.cc:2454] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22329153174443a387d8639d363c15ae in term 1.
I20260927 00:41:42.293448 4493 catalog_manager.cc:5495] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 22329153174443a387d8639d363c15ae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "22329153174443a387d8639d363c15ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22329153174443a387d8639d363c15ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:42.349700 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.346434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.0.98.1:41239)
0927 00:41:42.346667 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.346670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.346683 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:42.346778 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.346780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.346837 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.346928 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:42.347136 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.347639 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.348446 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:42.348654 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:42.348735 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:42.348980 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:42.349397 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:42.349487 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.349546 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.349594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260927 00:41:42.355222 4903 raft_consensus.cc:1261] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 1 FOLLOWER]: Refusing update from remote peer 22329153174443a387d8639d363c15ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:42.355222 4770 raft_consensus.cc:1261] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 1 FOLLOWER]: Refusing update from remote peer 22329153174443a387d8639d363c15ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:42.355871 4973 consensus_queue.cc:1035] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf883cf976400e9299fccba8e0134c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:42.356093 4977 consensus_queue.cc:1035] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7e125286ba444f8ea4e522ec5a3f1e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:42.371420 4493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:42.371565 4493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:42.374061 4493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc4e845d9bda47cc95fbc399b2f98559: Sending DeleteTablet for 3 replicas of tablet 529f8cb821654391bff661d9a0f3626d
I20260927 00:41:42.374812 4617 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f8cb821654391bff661d9a0f3626d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39074
I20260927 00:41:42.374876 4883 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f8cb821654391bff661d9a0f3626d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49788
I20260927 00:41:42.374918 4750 tablet_service.cc:1514] Processing DeleteTablet for tablet 529f8cb821654391bff661d9a0f3626d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38130
I20260927 00:41:42.375401 4991 tablet_replica.cc:330] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae: stopping tablet replica
I20260927 00:41:42.375680 4990 tablet_replica.cc:330] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: stopping tablet replica
I20260927 00:41:42.375707 4991 raft_consensus.cc:2229] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:42.376029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4555
I20260927 00:41:42.376077 4991 raft_consensus.cc:2258] T 529f8cb821654391bff661d9a0f3626d P 22329153174443a387d8639d363c15ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:42.376863 4992 tablet_replica.cc:330] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: stopping tablet replica
I20260927 00:41:42.377055 4990 raft_consensus.cc:2229] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:42.377223 4992 raft_consensus.cc:2229] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:42.377447 4990 raft_consensus.cc:2258] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:42.377481 4992 raft_consensus.cc:2258] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:42.378897 4990 ts_tablet_manager.cc:1905] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:42.379150 4992 ts_tablet_manager.cc:1905] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:42.381652 4990 ts_tablet_manager.cc:1918] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:42.381652 4992 ts_tablet_manager.cc:1918] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:42.381752 4992 log.cc:1196] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/529f8cb821654391bff661d9a0f3626d
I20260927 00:41:42.381752 4990 log.cc:1196] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/529f8cb821654391bff661d9a0f3626d
I20260927 00:41:42.382073 4992 ts_tablet_manager.cc:1939] T 529f8cb821654391bff661d9a0f3626d P a8cf883cf976400e9299fccba8e0134c: Deleting consensus metadata
I20260927 00:41:42.382083 4990 ts_tablet_manager.cc:1939] T 529f8cb821654391bff661d9a0f3626d P bf7e125286ba444f8ea4e522ec5a3f1e: Deleting consensus metadata
W20260927 00:41:42.382900 4478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41239 (error 108)
I20260927 00:41:42.383208 4480 catalog_manager.cc:4841] TS bf7e125286ba444f8ea4e522ec5a3f1e (127.0.98.3:33823): tablet 529f8cb821654391bff661d9a0f3626d (table test-table [id=b681288b2d1c41c6aa1cc3c181a97064]) successfully deleted
I20260927 00:41:42.383416 4480 catalog_manager.cc:4841] TS a8cf883cf976400e9299fccba8e0134c (127.0.98.2:43213): tablet 529f8cb821654391bff661d9a0f3626d (table test-table [id=b681288b2d1c41c6aa1cc3c181a97064]) successfully deleted
I20260927 00:41:42.383848 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.383596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41239 (local address 127.0.0.1:39084)
0927 00:41:42.383702 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.383767 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:41:42.383911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4688
W20260927 00:41:42.384060 4478 catalog_manager.cc:4584] TS 22329153174443a387d8639d363c15ae (127.0.98.1:41239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 529f8cb821654391bff661d9a0f3626d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41239: connect: Connection refused (error 111)
I20260927 00:41:42.389649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4821
I20260927 00:41:42.395109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4462
2026-09-27T00:41:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2586 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T00:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:42Z Disabled control of system clock
I20260927 00:41:42.425673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42719
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:41:42.532696 4997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:42.533015 4997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:42.533056 4997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:42.536919 4997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:42.536963 4997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:42.536998 4997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:42.537014 4997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:42.537029 4997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:42.541648 4997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:42.542579 4997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:42.543848 4997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:42.552094 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:42.552151 4997 server_base.cc:1034] running on GCE node
W20260927 00:41:42.552094 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:42.552271 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:42.552971 4997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:42.553957 4997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:42.555155 4997 hybrid_clock.cc:648] HybridClock initialized: now 1790469702555108 us; error 39 us; skew 500 ppm
I20260927 00:41:42.557125 4997 webserver.cc:460] Webserver started at http://127.0.98.62:40747/ using document root <none> and password file <none>
I20260927 00:41:42.557682 4997 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:42.557744 4997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:42.557966 4997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:42.559938 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "aa6341b3de5441b48719181baa3696fb"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "cb9f3e43970a1eb9c5583e0d7ceca0c3"
server_key_iv: "2812bd231787ad41d43c9d2b3dfc1197"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.560503 4997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa6341b3de5441b48719181baa3696fb"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "cb9f3e43970a1eb9c5583e0d7ceca0c3"
server_key_iv: "2812bd231787ad41d43c9d2b3dfc1197"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.564011 4997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:41:42.566303 5011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.567449 4997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:42.567579 4997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "aa6341b3de5441b48719181baa3696fb"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "cb9f3e43970a1eb9c5583e0d7ceca0c3"
server_key_iv: "2812bd231787ad41d43c9d2b3dfc1197"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.567684 4997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:42.572062 4997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:42.572613 4997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:42.572785 4997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:42.579336 4997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42719
I20260927 00:41:42.579337 5063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42719 every 8 connection(s)
I20260927 00:41:42.580226 4997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:42.583174 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:42.588325 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: Bootstrap starting.
I20260927 00:41:42.590425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 4997
I20260927 00:41:42.590538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:42.590778 392 external_mini_cluster.cc:1428] Setting key e1b51469bd203493ef72142756c68ae9
I20260927 00:41:42.592274 5064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:42.593122 5064 log.cc:824] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:42.595283 5064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: No bootstrap required, opened a new log
I20260927 00:41:42.599303 5064 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } }
I20260927 00:41:42.599534 5064 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:42.599593 5064 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6341b3de5441b48719181baa3696fb, State: Initialized, Role: FOLLOWER
I20260927 00:41:42.600036 5064 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } }
I20260927 00:41:42.600183 5064 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:42.600266 5064 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:42.600380 5064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:42.600721 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.595270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.98.62:42719)
0927 00:41:42.596418 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.596449 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.596490 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:41:42.597829 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.597849 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.598126 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.598447 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:42.599209 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:42.599214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:42.599225 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:42.599235 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:42.599287 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:42.599302 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:42.599411 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.599518 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.599778 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":129,"threads_started":1}
I20260927 00:41:42.601353 5064 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } }
I20260927 00:41:42.601672 5064 leader_election.cc:302] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6341b3de5441b48719181baa3696fb; no voters:
I20260927 00:41:42.601986 5064 leader_election.cc:288] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:42.602072 5069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:42.602238 5069 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [term 1 LEADER]: Becoming Leader. State: Replica: aa6341b3de5441b48719181baa3696fb, State: Running, Role: LEADER
I20260927 00:41:42.602519 5069 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } }
I20260927 00:41:42.603199 5064 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:42.604187 5070 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6341b3de5441b48719181baa3696fb. Latest consensus state: current_term: 1 leader_uuid: "aa6341b3de5441b48719181baa3696fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } } }
I20260927 00:41:42.604326 5070 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:42.604579 5071 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6341b3de5441b48719181baa3696fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6341b3de5441b48719181baa3696fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42719 } } }
I20260927 00:41:42.604655 5071 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:42.605647 5078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:42.608434 5078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:42.614027 5078 catalog_manager.cc:1347] Generated new cluster ID: dcdb022aac5148f78769f87452850a2c
I20260927 00:41:42.614108 5078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:42.632339 5078 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:42.632694 5078 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:42.648087 5078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: Generated new TSK 0
I20260927 00:41:42.648943 5078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:42.657023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--builtin_ntp_servers=127.0.98.20:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:41:42.764931 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:42.765161 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:42.765199 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:42.769004 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:42.769043 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:42.769096 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:42.773682 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:42.774580 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:42.775852 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:42.785111 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:42.785146 5088 server_base.cc:1034] running on GCE node
W20260927 00:41:42.785099 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:42.785324 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:42.785677 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:42.786269 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:42.787417 5088 hybrid_clock.cc:648] HybridClock initialized: now 1790469702787384 us; error 48 us; skew 500 ppm
I20260927 00:41:42.789172 5088 webserver.cc:460] Webserver started at http://127.0.98.1:45301/ using document root <none> and password file <none>
I20260927 00:41:42.789703 5088 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:42.789762 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:42.789994 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:42.791890 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "592cde31574f4ccf9b2caacc79f273b4"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "de7e7a42d5efc3e58e1181a5e2a7d834"
server_key_iv: "a10f367df8c89ec025568212c1adade3"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.792421 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "592cde31574f4ccf9b2caacc79f273b4"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "de7e7a42d5efc3e58e1181a5e2a7d834"
server_key_iv: "a10f367df8c89ec025568212c1adade3"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.795874 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260927 00:41:42.798161 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.799285 5088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:42.799405 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "592cde31574f4ccf9b2caacc79f273b4"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "de7e7a42d5efc3e58e1181a5e2a7d834"
server_key_iv: "a10f367df8c89ec025568212c1adade3"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.799506 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:42.803849 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:42.804380 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:42.804589 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:42.805145 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:42.805958 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:42.806022 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.806082 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:42.806128 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.814193 5088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36083
I20260927 00:41:42.814215 5215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36083 every 8 connection(s)
I20260927 00:41:42.814980 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:42.820631 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.816589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55153 (local address 127.0.98.62:42719)
0927 00:41:42.816711 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.816714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.817689 (+ 975us) server_negotiation.cc:407] Connection header received
0927 00:41:42.818984 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.818988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.819032 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.819134 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:42.820146 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:42.820149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:42.820152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:42.820164 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:42.820190 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:42.820198 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:42.820301 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.820432 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.820503 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:41:42.821455 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.816970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42719 (local address 127.0.98.1:55153)
0927 00:41:42.817554 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.817591 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.818788 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.819289 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.819295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.819388 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:42.819882 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:42.819883 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:42.819938 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:42.820008 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:42.820286 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:42.820294 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:42.820299 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.820480 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.820699 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260927 00:41:42.822454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5088
I20260927 00:41:42.822479 5216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42719
I20260927 00:41:42.822592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:42.822690 5216 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:42.822898 392 external_mini_cluster.cc:1428] Setting key f4545068ffc5e9cfa43bab8fc88df21e
I20260927 00:41:42.823130 5216 heartbeater.cc:511] Master 127.0.98.62:42719 requested a full tablet report, sending...
I20260927 00:41:42.824787 5027 ts_manager.cc:194] Registered new tserver with Master: 592cde31574f4ccf9b2caacc79f273b4 (127.0.98.1:36083)
I20260927 00:41:42.824826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--builtin_ntp_servers=127.0.98.20:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:41:42.932209 5219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:42.932449 5219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:42.932487 5219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:42.936406 5219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:42.936460 5219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:42.936546 5219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:42.941196 5219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:42.942162 5219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:42.943440 5219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:42.952428 5224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:42.952661 5225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:42.952626 5227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:42.952642 5219 server_base.cc:1034] running on GCE node
I20260927 00:41:42.953145 5219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:42.953711 5219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:42.954855 5219 hybrid_clock.cc:648] HybridClock initialized: now 1790469702954826 us; error 46 us; skew 500 ppm
I20260927 00:41:42.956674 5219 webserver.cc:460] Webserver started at http://127.0.98.2:45249/ using document root <none> and password file <none>
I20260927 00:41:42.957247 5219 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:42.957309 5219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:42.957520 5219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:42.959421 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "edc5ad64c3b345a3953dc0065b761b08"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "70b4ac88e8a45ab262cbc9188e36dc79"
server_key_iv: "686a60a8b3b31cb1ddc9bc4288646309"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.959959 5219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "edc5ad64c3b345a3953dc0065b761b08"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "70b4ac88e8a45ab262cbc9188e36dc79"
server_key_iv: "686a60a8b3b31cb1ddc9bc4288646309"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.963436 5219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:41:42.965655 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.966801 5219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:41:42.966909 5219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "edc5ad64c3b345a3953dc0065b761b08"
format_stamp: "Formatted at 2026-09-27 00:41:42 on dist-test-slave-pkws"
server_key: "70b4ac88e8a45ab262cbc9188e36dc79"
server_key_iv: "686a60a8b3b31cb1ddc9bc4288646309"
server_key_version: "encryptionkey@0"
I20260927 00:41:42.967012 5219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:42.971454 5219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:42.972033 5219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:42.972198 5219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:42.972697 5219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:42.973594 5219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:42.973640 5219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.973702 5219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:42.973737 5219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:42.982612 5219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43153
I20260927 00:41:42.982637 5346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43153 every 8 connection(s)
I20260927 00:41:42.983419 5219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:42.989235 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.985139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38091 (local address 127.0.98.62:42719)
0927 00:41:42.985269 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:42.985273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:42.986278 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 00:41:42.987592 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:42.987595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:42.987648 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:42.987734 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:42.988761 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:42.988764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:42.988767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:42.988779 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:42.988799 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:42.988807 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:42.988926 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:42.989060 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:41:42.989103 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:42.990109 5349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:42.985468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42719 (local address 127.0.98.2:38091)
0927 00:41:42.986083 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:42.986120 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:42.987387 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:42.987924 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:42.987930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:42.988017 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:42.988523 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:42.988524 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:42.988561 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:42.988626 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:42.988912 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:42.988920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:42.988925 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:41:42.989103 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:41:42.989399 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260927 00:41:42.990619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5219
I20260927 00:41:42.990759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:42.991075 392 external_mini_cluster.cc:1428] Setting key 5a9e86a2c28e709848e1e332a41cf653
I20260927 00:41:42.991219 5347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42719
I20260927 00:41:42.991451 5347 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:42.991884 5347 heartbeater.cc:511] Master 127.0.98.62:42719 requested a full tablet report, sending...
I20260927 00:41:42.992864 5027 ts_manager.cc:194] Registered new tserver with Master: edc5ad64c3b345a3953dc0065b761b08 (127.0.98.2:43153)
I20260927 00:41:42.993196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--builtin_ntp_servers=127.0.98.20:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 00:41:43.100121 5350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:43.100349 5350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:43.100389 5350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:43.104944 5350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:43.104991 5350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:43.105046 5350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:43.110309 5350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:43.111202 5350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:43.112599 5350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:43.121721 5355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:43.121721 5356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:43.121784 5358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:43.122051 5350 server_base.cc:1034] running on GCE node
I20260927 00:41:43.122463 5350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:43.123035 5350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:43.124194 5350 hybrid_clock.cc:648] HybridClock initialized: now 1790469703124184 us; error 27 us; skew 500 ppm
I20260927 00:41:43.125936 5350 webserver.cc:460] Webserver started at http://127.0.98.3:41649/ using document root <none> and password file <none>
I20260927 00:41:43.126438 5350 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:43.126513 5350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:43.126669 5350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:43.128630 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "47201cb294024a50998c6c056d699fac"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "e76000fdab018b88cc8a5e787166ee15"
server_key_iv: "96b18fb83f2c2706dc00d0c0aa2d47cf"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.129189 5350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47201cb294024a50998c6c056d699fac"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "e76000fdab018b88cc8a5e787166ee15"
server_key_iv: "96b18fb83f2c2706dc00d0c0aa2d47cf"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.132591 5350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:43.134914 5364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:43.136082 5350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:43.136181 5350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "47201cb294024a50998c6c056d699fac"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "e76000fdab018b88cc8a5e787166ee15"
server_key_iv: "96b18fb83f2c2706dc00d0c0aa2d47cf"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.136263 5350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:43.140591 5350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:43.141220 5350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:43.141376 5350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:43.141851 5350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:43.142666 5350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:43.142711 5350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:43.142747 5350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:43.142762 5350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:43.151226 5350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40079
I20260927 00:41:43.151247 5477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40079 every 8 connection(s)
I20260927 00:41:43.152076 5350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:43.158270 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.154127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35731 (local address 127.0.98.62:42719)
0927 00:41:43.154305 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.154309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.155299 (+ 990us) server_negotiation.cc:407] Connection header received
0927 00:41:43.156623 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.156627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.156670 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.156746 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.157812 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.157824 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.157826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.157838 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.157859 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.157873 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.157967 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.158096 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.158138 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:41:43.158851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5350
I20260927 00:41:43.158970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:43.159250 392 external_mini_cluster.cc:1428] Setting key cd4a2ad7812ba1a2e6a074525b4cc43f
I20260927 00:41:43.159372 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.154519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42719 (local address 127.0.98.3:35731)
0927 00:41:43.155144 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.155192 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.156414 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.156991 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.156998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.157084 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.157573 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.157575 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.157612 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.157677 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.157952 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.157960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.157965 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.158139 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.158415 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260927 00:41:43.161036 5478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42719
I20260927 00:41:43.161258 5478 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:43.161710 5478 heartbeater.cc:511] Master 127.0.98.62:42719 requested a full tablet report, sending...
I20260927 00:41:43.162604 5027 ts_manager.cc:194] Registered new tserver with Master: 47201cb294024a50998c6c056d699fac (127.0.98.3:40079)
I20260927 00:41:43.173172 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:43.176566 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.174960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.0.98.62:42719)
0927 00:41:43.175099 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.175104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.175248 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:41:43.175401 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.175405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.175442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.175521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.176140 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.176143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.176146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.176156 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.176174 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.176183 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.176272 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.176402 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.176444 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:41:43.179189 5027 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:43.182230 5027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:43.193081 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.189863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40079 (local address 127.0.0.1:50428)
0927 00:41:43.190382 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.190428 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.190516 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.191067 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.191073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.191154 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.191657 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.191659 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.191680 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.191754 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.192722 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.192730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.192735 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.192856 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.192943 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":69,"threads_started":1}
I20260927 00:41:43.193209 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.190025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43153 (local address 127.0.0.1:42598)
0927 00:41:43.190686 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.190698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.190759 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.191689 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.191692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.191706 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.192121 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.192123 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.192134 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.192159 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.192896 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.192919 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.192922 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.192977 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.193012 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":109,"threads_started":1}
I20260927 00:41:43.193358 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.190554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36083 (local address 127.0.0.1:59492)
0927 00:41:43.190935 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.190947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.191013 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.192348 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.192351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.192364 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.192725 (+ 361us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.192727 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.192736 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.192758 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.193154 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.193157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.193159 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.193228 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.193259 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":86,"threads_started":1}
I20260927 00:41:43.193120 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.189991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.98.3:40079)
0927 00:41:43.190584 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.190598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.190624 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:41:43.190683 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.190691 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.190882 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.191427 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.192364 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.192368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.192376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.192383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.192413 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.192427 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.192563 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.192872 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.192973 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":145,"threads_started":1}
I20260927 00:41:43.193619 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.190563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.0.98.2:43153)
0927 00:41:43.191258 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.191270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.191293 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:41:43.191356 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.191363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.191535 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.191917 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.192711 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.192715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.192726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.192736 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.192770 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.192798 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.193357 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.193405 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.193471 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":122,"threads_started":1}
I20260927 00:41:43.193708 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.190953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.98.1:36083)
0927 00:41:43.191366 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.191377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.191396 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:41:43.191440 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.191444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.191572 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.191690 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.192923 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.192930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.192941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.192956 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.192998 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.193025 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.193116 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.193492 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.193549 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":128,"threads_started":1}
I20260927 00:41:43.196117 5281 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 (DEFAULT_TABLE table=test-table [id=538306c7d0d24f2ab75ebe9cdc02061a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:43.196127 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 (DEFAULT_TABLE table=test-table [id=538306c7d0d24f2ab75ebe9cdc02061a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:43.197098 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3e66cd9894a1a9f586b8eeeaa3243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:43.197108 5281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3e66cd9894a1a9f586b8eeeaa3243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:43.198330 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 (DEFAULT_TABLE table=test-table [id=538306c7d0d24f2ab75ebe9cdc02061a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:43.199222 5150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da3e66cd9894a1a9f586b8eeeaa3243. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:43.202986 5495 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Bootstrap starting.
I20260927 00:41:43.204206 5496 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Bootstrap starting.
I20260927 00:41:43.205950 5495 tablet_bootstrap.cc:654] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:43.205948 5497 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Bootstrap starting.
I20260927 00:41:43.206526 5496 tablet_bootstrap.cc:654] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:43.206936 5495 log.cc:824] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:43.207332 5496 log.cc:824] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:43.207806 5497 tablet_bootstrap.cc:654] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:43.209561 5497 log.cc:824] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:43.210523 5495 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: No bootstrap required, opened a new log
I20260927 00:41:43.210551 5496 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: No bootstrap required, opened a new log
I20260927 00:41:43.210721 5495 ts_tablet_manager.cc:1397] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 00:41:43.210726 5496 ts_tablet_manager.cc:1397] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:41:43.211737 5497 tablet_bootstrap.cc:492] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: No bootstrap required, opened a new log
I20260927 00:41:43.211905 5497 ts_tablet_manager.cc:1397] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260927 00:41:43.214146 5495 raft_consensus.cc:348] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.214386 5495 raft_consensus.cc:374] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:43.214447 5495 raft_consensus.cc:729] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47201cb294024a50998c6c056d699fac, State: Initialized, Role: FOLLOWER
I20260927 00:41:43.214883 5495 consensus_queue.cc:260] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.215147 5496 raft_consensus.cc:348] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.215376 5496 raft_consensus.cc:374] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:43.215431 5496 raft_consensus.cc:729] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc5ad64c3b345a3953dc0065b761b08, State: Initialized, Role: FOLLOWER
I20260927 00:41:43.215878 5495 ts_tablet_manager.cc:1428] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260927 00:41:43.215848 5496 consensus_queue.cc:260] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.216245 5478 heartbeater.cc:503] Master 127.0.98.62:42719 was elected leader, sending a full tablet report...
I20260927 00:41:43.216255 5497 raft_consensus.cc:348] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.216492 5497 raft_consensus.cc:374] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:43.216544 5497 raft_consensus.cc:729] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592cde31574f4ccf9b2caacc79f273b4, State: Initialized, Role: FOLLOWER
I20260927 00:41:43.216748 5496 ts_tablet_manager.cc:1428] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:43.216949 5497 consensus_queue.cc:260] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.217072 5347 heartbeater.cc:503] Master 127.0.98.62:42719 was elected leader, sending a full tablet report...
I20260927 00:41:43.217731 5497 ts_tablet_manager.cc:1428] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:41:43.217991 5216 heartbeater.cc:503] Master 127.0.98.62:42719 was elected leader, sending a full tablet report...
W20260927 00:41:43.234349 5348 tablet.cc:2369] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:41:43.316090 5217 tablet.cc:2369] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:41:43.403026 5479 tablet.cc:2369] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:43.602484 5503 raft_consensus.cc:482] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:43.602732 5503 raft_consensus.cc:504] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.604003 5503 leader_election.cc:288] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47201cb294024a50998c6c056d699fac (127.0.98.3:40079), edc5ad64c3b345a3953dc0065b761b08 (127.0.98.2:43153)
I20260927 00:41:43.606290 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.604458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47177 (local address 127.0.98.3:40079)
0927 00:41:43.604688 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.604691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.604709 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:41:43.604929 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.604933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.604973 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.605065 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.605889 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.605895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.605897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.605910 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.605933 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.605943 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.606053 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.606130 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.606164 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260927 00:41:43.606289 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.604422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53299 (local address 127.0.98.2:43153)
0927 00:41:43.604604 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.604608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.604865 (+ 257us) server_negotiation.cc:407] Connection header received
0927 00:41:43.604936 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.604939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.604973 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.605065 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.605919 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.605922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.605924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.605934 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.605952 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.605964 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.606044 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.606123 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.606164 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 00:41:43.606348 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.604231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40079 (local address 127.0.98.1:47177)
0927 00:41:43.604646 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.604661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.604759 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.605094 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.605097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.605113 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.605687 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.605689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.605712 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.605747 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.606044 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.606047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.606050 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.606127 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.606190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":18,"thread_start_us":116,"threads_started":1}
I20260927 00:41:43.606709 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.604341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43153 (local address 127.0.98.1:53299)
0927 00:41:43.604726 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.604738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.604831 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.605094 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.605097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.605113 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:43.605750 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:41:43.605752 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:43.605759 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:41:43.605781 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:43.606045 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:43.606047 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:43.606050 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:41:43.606127 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:41:43.606178 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":73,"threads_started":1}
I20260927 00:41:43.606985 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3e66cd9894a1a9f586b8eeeaa3243" candidate_uuid: "592cde31574f4ccf9b2caacc79f273b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47201cb294024a50998c6c056d699fac" is_pre_election: true
I20260927 00:41:43.607259 5432 raft_consensus.cc:2454] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592cde31574f4ccf9b2caacc79f273b4 in term 0.
I20260927 00:41:43.607280 5301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3e66cd9894a1a9f586b8eeeaa3243" candidate_uuid: "592cde31574f4ccf9b2caacc79f273b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5ad64c3b345a3953dc0065b761b08" is_pre_election: true
I20260927 00:41:43.607605 5301 raft_consensus.cc:2454] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592cde31574f4ccf9b2caacc79f273b4 in term 0.
I20260927 00:41:43.607750 5105 leader_election.cc:302] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47201cb294024a50998c6c056d699fac, 592cde31574f4ccf9b2caacc79f273b4; no voters:
I20260927 00:41:43.608006 5503 raft_consensus.cc:2781] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:43.608094 5503 raft_consensus.cc:482] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:43.608150 5503 raft_consensus.cc:3037] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:43.609220 5503 raft_consensus.cc:504] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.609510 5503 leader_election.cc:288] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [CANDIDATE]: Term 1 election: Requested vote from peers 47201cb294024a50998c6c056d699fac (127.0.98.3:40079), edc5ad64c3b345a3953dc0065b761b08 (127.0.98.2:43153)
I20260927 00:41:43.609789 5432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3e66cd9894a1a9f586b8eeeaa3243" candidate_uuid: "592cde31574f4ccf9b2caacc79f273b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47201cb294024a50998c6c056d699fac"
I20260927 00:41:43.609838 5301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da3e66cd9894a1a9f586b8eeeaa3243" candidate_uuid: "592cde31574f4ccf9b2caacc79f273b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc5ad64c3b345a3953dc0065b761b08"
I20260927 00:41:43.609910 5432 raft_consensus.cc:3037] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:43.609953 5301 raft_consensus.cc:3037] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:43.610864 5301 raft_consensus.cc:2454] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592cde31574f4ccf9b2caacc79f273b4 in term 1.
I20260927 00:41:43.610961 5432 raft_consensus.cc:2454] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592cde31574f4ccf9b2caacc79f273b4 in term 1.
I20260927 00:41:43.611136 5104 leader_election.cc:302] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592cde31574f4ccf9b2caacc79f273b4, edc5ad64c3b345a3953dc0065b761b08; no voters:
I20260927 00:41:43.611332 5503 raft_consensus.cc:2781] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:43.611593 5503 raft_consensus.cc:686] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [term 1 LEADER]: Becoming Leader. State: Replica: 592cde31574f4ccf9b2caacc79f273b4, State: Running, Role: LEADER
I20260927 00:41:43.611891 5503 consensus_queue.cc:237] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } }
I20260927 00:41:43.615442 5028 catalog_manager.cc:5495] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 592cde31574f4ccf9b2caacc79f273b4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "592cde31574f4ccf9b2caacc79f273b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592cde31574f4ccf9b2caacc79f273b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36083 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:43.648383 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.647075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.0.98.1:36083)
0927 00:41:43.647192 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.647195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.647207 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:43.647295 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.647297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.647387 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.647461 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.648065 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.648068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.648070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.648081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.648098 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.648105 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.648177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.648219 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.648250 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:41:43.653963 5301 raft_consensus.cc:1261] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 1 FOLLOWER]: Refusing update from remote peer 592cde31574f4ccf9b2caacc79f273b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:43.653963 5432 raft_consensus.cc:1261] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 1 FOLLOWER]: Refusing update from remote peer 592cde31574f4ccf9b2caacc79f273b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:43.654712 5506 consensus_queue.cc:1035] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47201cb294024a50998c6c056d699fac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:43.654918 5503 consensus_queue.cc:1035] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc5ad64c3b345a3953dc0065b761b08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:43.662662 5511 mvcc.cc:205] Tried to move back new op lower bound from 7333763906159562752 to 7333763905999093760. Current Snapshot: MvccSnapshot[applied={T|T < 7333763906159562752}]
I20260927 00:41:43.669860 5026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:43.670037 5026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:43.672302 5026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa6341b3de5441b48719181baa3696fb: Sending DeleteTablet for 3 replicas of tablet 4da3e66cd9894a1a9f586b8eeeaa3243
I20260927 00:41:43.673043 5281 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:43 UTC) from {username='slave'} at 127.0.0.1:42598
I20260927 00:41:43.673065 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:43 UTC) from {username='slave'} at 127.0.0.1:59492
I20260927 00:41:43.673501 5520 tablet_replica.cc:330] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: stopping tablet replica
I20260927 00:41:43.673528 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da3e66cd9894a1a9f586b8eeeaa3243 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:43 UTC) from {username='slave'} at 127.0.0.1:50428
I20260927 00:41:43.673887 5520 raft_consensus.cc:2229] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:43.674002 5521 tablet_replica.cc:330] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: stopping tablet replica
I20260927 00:41:43.674261 5520 raft_consensus.cc:2258] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:43.674306 5521 raft_consensus.cc:2229] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:43.674571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5088
I20260927 00:41:43.674764 5519 tablet_replica.cc:330] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 592cde31574f4ccf9b2caacc79f273b4: stopping tablet replica
I20260927 00:41:43.675339 5521 raft_consensus.cc:2258] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:43.676590 5520 ts_tablet_manager.cc:1905] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:43.676787 5521 ts_tablet_manager.cc:1905] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:43.679224 5520 ts_tablet_manager.cc:1918] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:43.679338 5520 log.cc:1196] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/4da3e66cd9894a1a9f586b8eeeaa3243
I20260927 00:41:43.679648 5520 ts_tablet_manager.cc:1939] T 4da3e66cd9894a1a9f586b8eeeaa3243 P edc5ad64c3b345a3953dc0065b761b08: Deleting consensus metadata
I20260927 00:41:43.680207 5521 ts_tablet_manager.cc:1918] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:43.680302 5521 log.cc:1196] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/4da3e66cd9894a1a9f586b8eeeaa3243
I20260927 00:41:43.680616 5521 ts_tablet_manager.cc:1939] T 4da3e66cd9894a1a9f586b8eeeaa3243 P 47201cb294024a50998c6c056d699fac: Deleting consensus metadata
I20260927 00:41:43.680734 5013 catalog_manager.cc:4841] TS edc5ad64c3b345a3953dc0065b761b08 (127.0.98.2:43153): tablet 4da3e66cd9894a1a9f586b8eeeaa3243 (table test-table [id=538306c7d0d24f2ab75ebe9cdc02061a]) successfully deleted
I20260927 00:41:43.681864 5014 catalog_manager.cc:4841] TS 47201cb294024a50998c6c056d699fac (127.0.98.3:40079): tablet 4da3e66cd9894a1a9f586b8eeeaa3243 (table test-table [id=538306c7d0d24f2ab75ebe9cdc02061a]) successfully deleted
W20260927 00:41:43.682802 5014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36083 (error 108)
I20260927 00:41:43.682868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5219
I20260927 00:41:43.685639 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.685379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36083 (local address 127.0.0.1:45414)
0927 00:41:43.685508 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.685567 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:41:43.685792 5014 catalog_manager.cc:4584] TS 592cde31574f4ccf9b2caacc79f273b4 (127.0.98.1:36083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4da3e66cd9894a1a9f586b8eeeaa3243: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36083: connect: Connection refused (error 111)
I20260927 00:41:43.688874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5350
I20260927 00:41:43.694237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 4997
2026-09-27T00:41:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1297 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 00:41:43.715317 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:41:43.716861 392 webserver.cc:460] Webserver started at https://127.0.0.1:42217/ using document root <none> and password file <none>
I20260927 00:41:43.717273 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:41:43.717561 392 webserver.cc:460] Webserver started at http://127.0.0.1:39293/ using document root <none> and password file <none>
2026-09-27T00:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:43Z Disabled control of system clock
I20260927 00:41:43.726158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34751
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34751
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/testchainca.cert with env {}
W20260927 00:41:43.836656 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:43.836959 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:43.837013 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:43.841012 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:43.841059 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:43.841075 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:43.841091 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:43.841106 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:41:43.841122 5528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:41:43.841136 5528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42217/jwks.json
I20260927 00:41:43.846517 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:43.847481 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:43.848392 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:43.856943 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:43.857015 5528 server_base.cc:1034] running on GCE node
W20260927 00:41:43.856943 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:43.856966 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:43.857766 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:43.858789 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:43.859946 5528 hybrid_clock.cc:648] HybridClock initialized: now 1790469703859913 us; error 46 us; skew 500 ppm
I20260927 00:41:43.861785 5528 webserver.cc:460] Webserver started at http://127.0.98.62:42113/ using document root <none> and password file <none>
I20260927 00:41:43.862340 5528 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:43.862399 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:43.862622 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:43.864552 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "3085478b208449af8e168c80a5201af0"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "ec29ccaec601594216fe09473b9b8dbd"
server_key_iv: "6e2abb6d0d7ace1438746825c16d4287"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.865131 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "3085478b208449af8e168c80a5201af0"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "ec29ccaec601594216fe09473b9b8dbd"
server_key_iv: "6e2abb6d0d7ace1438746825c16d4287"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.868558 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 00:41:43.870949 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:43.872215 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:43.872363 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "3085478b208449af8e168c80a5201af0"
format_stamp: "Formatted at 2026-09-27 00:41:43 on dist-test-slave-pkws"
server_key: "ec29ccaec601594216fe09473b9b8dbd"
server_key_iv: "6e2abb6d0d7ace1438746825c16d4287"
server_key_version: "encryptionkey@0"
I20260927 00:41:43.872484 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:43.885370 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:43.886029 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:43.886217 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:43.893086 5528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34751
I20260927 00:41:43.893115 5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34751 every 8 connection(s)
I20260927 00:41:43.893990 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:43.897187 5595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:43.901289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5528
I20260927 00:41:43.901432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:43.901721 392 external_mini_cluster.cc:1428] Setting key c603e684ec2b73683cd4236d11b1a797
I20260927 00:41:43.902738 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0: Bootstrap starting.
I20260927 00:41:43.904827 5595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:43.905558 5595 log.cc:824] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:43.907248 5595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0: No bootstrap required, opened a new log
I20260927 00:41:43.910590 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.903074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.0.98.62:34751)
0927 00:41:43.903668 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.903681 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.903715 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:41:43.904315 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.904331 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.904712 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.905112 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:43.905540 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.906510 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.907408 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.907858 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.908027 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:43.908928 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:43.908939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:43.908951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:43.909308 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:43.909389 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:43.909411 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:43.909522 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:43.909675 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:41:43.909912 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
I20260927 00:41:43.910980 5595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } }
I20260927 00:41:43.911244 5595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:43.911300 5595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3085478b208449af8e168c80a5201af0, State: Initialized, Role: FOLLOWER
I20260927 00:41:43.911723 5595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } }
I20260927 00:41:43.911844 5595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:43.911901 5595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:43.911998 5595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:43.913062 5595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } }
I20260927 00:41:43.913427 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:43.913429 5595 leader_election.cc:302] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3085478b208449af8e168c80a5201af0; no voters:
I20260927 00:41:43.913733 5595 leader_election.cc:288] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:43.913838 5600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:43.914058 5600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [term 1 LEADER]: Becoming Leader. State: Replica: 3085478b208449af8e168c80a5201af0, State: Running, Role: LEADER
I20260927 00:41:43.914359 5600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } }
I20260927 00:41:43.914921 5595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:43.916074 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3085478b208449af8e168c80a5201af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } } }
I20260927 00:41:43.916095 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3085478b208449af8e168c80a5201af0. Latest consensus state: current_term: 1 leader_uuid: "3085478b208449af8e168c80a5201af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3085478b208449af8e168c80a5201af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34751 } } }
I20260927 00:41:43.916214 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:43.916214 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:43.916600 5609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:43.919493 5609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:43.924561 5609 catalog_manager.cc:1347] Generated new cluster ID: a83992719ed74ceda22f9ef5ffeb4bdd
I20260927 00:41:43.924639 5609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:43.933046 5609 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:43.934231 5609 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:43.942914 5609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3085478b208449af8e168c80a5201af0: Generated new TSK 0
I20260927 00:41:43.943665 5609 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:41:43.974370 5625 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 00:41:43.974483 5625 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:41:43.974953 5598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:43.958129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.98.62:34751)
0927 00:41:43.958255 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:43.958260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:43.958456 (+ 196us) server_negotiation.cc:407] Connection header received
0927 00:41:43.958654 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:43.958656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:43.958697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:43.958779 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:41:43.959050 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.959552 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.960738 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.961010 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.961085 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:43.961462 (+ 377us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:41:43.974642 (+ 13180us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:41:43.974843 (+ 201us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:41:43.975044 5598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 00:41:43.976372 5624 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:43.958054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34751 (local address 127.0.0.1:36480)
0927 00:41:43.958373 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:43.958388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:43.958513 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:43.958812 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:43.958816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:43.958824 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:41:43.958950 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:43.958956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.959649 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.959653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:43.960630 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:43.960640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:43.961096 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:43.961114 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:43.961216 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:43.961263 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:41:43.974836 (+ 13573us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 00:41:43.975071 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 00:41:43.977062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5528
2026-09-27T00:41:43Z chronyd exiting
I20260927 00:41:44.014284 392 test_util.cc:175] -----------------------------------------------
I20260927 00:41:44.014371 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790469678562534-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 00:41:44.015347 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:41:44.016148 392 webserver.cc:460] Webserver started at https://127.0.0.1:42813/ using document root <none> and password file <none>
I20260927 00:41:44.016225 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:41:44.016466 392 webserver.cc:460] Webserver started at http://127.0.0.1:32985/ using document root <none> and password file <none>
2026-09-27T00:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:44Z Disabled control of system clock
I20260927 00:41:44.025044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35273
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42813/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/testchainca.cert with env {}
W20260927 00:41:44.130822 5631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:44.131125 5631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:44.131207 5631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:44.135167 5631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:44.135250 5631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:44.135279 5631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:44.135315 5631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:44.135355 5631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:41:44.135390 5631 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:41:44.135421 5631 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42813/jwks.json
I20260927 00:41:44.140568 5631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42813/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:44.141592 5631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:44.142555 5631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:44.150725 5636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:44.150808 5639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:44.150695 5637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:44.150942 5631 server_base.cc:1034] running on GCE node
I20260927 00:41:44.151491 5631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:44.152524 5631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:44.153736 5631 hybrid_clock.cc:648] HybridClock initialized: now 1790469704153721 us; error 28 us; skew 500 ppm
I20260927 00:41:44.155701 5631 webserver.cc:460] Webserver started at http://127.0.98.62:39051/ using document root <none> and password file <none>
I20260927 00:41:44.156245 5631 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:44.156332 5631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:44.156543 5631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:44.158525 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "67a58c5a33c94e59a8e99510034bcfc4"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "b208b62e5fbefda8917b9debe36b3c8e"
server_key_iv: "96c14b651090c56fb8a1ada34725f669"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.159068 5631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "67a58c5a33c94e59a8e99510034bcfc4"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "b208b62e5fbefda8917b9debe36b3c8e"
server_key_iv: "96c14b651090c56fb8a1ada34725f669"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.162616 5631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:41:44.164995 5645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:44.166224 5631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:41:44.166381 5631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "67a58c5a33c94e59a8e99510034bcfc4"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "b208b62e5fbefda8917b9debe36b3c8e"
server_key_iv: "96c14b651090c56fb8a1ada34725f669"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.166489 5631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:44.176820 5631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:44.177415 5631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:44.177589 5631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:44.184139 5631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35273
I20260927 00:41:44.184144 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35273 every 8 connection(s)
I20260927 00:41:44.185097 5631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:44.188042 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:44.189937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5631
I20260927 00:41:44.190078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:44.190359 392 external_mini_cluster.cc:1428] Setting key 98229c047594d782bb51b7c1c94116a4
I20260927 00:41:44.193761 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4: Bootstrap starting.
I20260927 00:41:44.195803 5698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:44.196699 5698 log.cc:824] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:44.198613 5698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4: No bootstrap required, opened a new log
I20260927 00:41:44.199385 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.191605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.0.98.62:35273)
0927 00:41:44.192318 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.192336 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.192370 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:41:44.193064 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.193082 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.193456 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.193782 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:44.194191 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.195104 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.196028 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.196740 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.196927 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.197766 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:44.197771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:44.197783 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:44.198141 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:44.198194 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:44.198213 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:44.198321 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:44.198460 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 00:41:44.198732 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":127,"threads_started":1}
I20260927 00:41:44.202000 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:44.202208 5698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } }
I20260927 00:41:44.202409 5698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:44.202477 5698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a58c5a33c94e59a8e99510034bcfc4, State: Initialized, Role: FOLLOWER
I20260927 00:41:44.202900 5698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } }
I20260927 00:41:44.203052 5698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:44.203131 5698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:44.203204 5698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:44.204079 5698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } }
I20260927 00:41:44.204402 5698 leader_election.cc:302] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a58c5a33c94e59a8e99510034bcfc4; no voters:
I20260927 00:41:44.204669 5698 leader_election.cc:288] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:44.204766 5703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:44.205017 5703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [term 1 LEADER]: Becoming Leader. State: Replica: 67a58c5a33c94e59a8e99510034bcfc4, State: Running, Role: LEADER
I20260927 00:41:44.205333 5703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } }
I20260927 00:41:44.205789 5698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:44.206822 5704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a58c5a33c94e59a8e99510034bcfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } } }
I20260927 00:41:44.206952 5704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.207221 5705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a58c5a33c94e59a8e99510034bcfc4. Latest consensus state: current_term: 1 leader_uuid: "67a58c5a33c94e59a8e99510034bcfc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a58c5a33c94e59a8e99510034bcfc4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35273 } } }
I20260927 00:41:44.207319 5705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.207394 5712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:44.210237 5712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:44.214963 5712 catalog_manager.cc:1347] Generated new cluster ID: b4de8915b8a34e94a4f5b665e04adf8f
I20260927 00:41:44.215022 5712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:44.231495 5712 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:44.232373 5712 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:44.242908 5712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a58c5a33c94e59a8e99510034bcfc4: Generated new TSK 0
I20260927 00:41:44.243574 5712 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 00:41:44.265300 5728 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 00:41:44.265373 5728 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 00:41:44.266038 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.250416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.62:35273)
0927 00:41:44.250529 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.250534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.250711 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:41:44.250890 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.250892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.250937 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.251043 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:41:44.251276 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.251802 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.252670 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.252942 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.253046 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.253339 (+ 293us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:41:44.265650 (+ 12311us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:41:44.265903 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50794: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 00:41:44.266103 5701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50794: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 00:41:44.266121 5727 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:44.250342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35273 (local address 127.0.0.1:50794)
0927 00:41:44.250631 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.250642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:44.250758 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:44.251046 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:44.251049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:44.251064 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:41:44.251169 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.251180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.251922 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.251925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.252550 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.252558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.253036 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.253060 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.253160 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.253180 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:41:44.265812 (+ 12632us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 00:41:44.265951 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42813/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
I20260927 00:41:44.266752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5631
2026-09-27T00:41:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (290 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 00:41:44.305521 392 mini_oidc.cc:152] Starting JWKS server
I20260927 00:41:44.306304 392 webserver.cc:460] Webserver started at https://127.0.0.1:42623/ using document root <none> and password file <none>
I20260927 00:41:44.306382 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 00:41:44.306584 392 webserver.cc:460] Webserver started at http://127.0.0.1:33189/ using document root <none> and password file <none>
2026-09-27T00:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:44Z Disabled control of system clock
I20260927 00:41:44.315178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37051
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:41:44.422231 5734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:44.422621 5734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:44.422713 5734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:44.427421 5734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:44.427492 5734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:44.427515 5734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:44.427548 5734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:44.427577 5734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:41:44.427599 5734 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:41:44.427618 5734 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42623/jwks.json
I20260927 00:41:44.433357 5734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:44.434391 5734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:44.435747 5734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:44.443872 5739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:44.443888 5740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:44.443876 5742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:44.444177 5734 server_base.cc:1034] running on GCE node
I20260927 00:41:44.444752 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:44.445967 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:44.447166 5734 hybrid_clock.cc:648] HybridClock initialized: now 1790469704447151 us; error 27 us; skew 500 ppm
I20260927 00:41:44.448975 5734 webserver.cc:460] Webserver started at http://127.0.98.62:43283/ using document root <none> and password file <none>
I20260927 00:41:44.449566 5734 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:44.449656 5734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:44.449867 5734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:44.451889 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "3da81647a548431aba7dc87cd9762b67"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "e59d7e9ef81c6694d4b3c9b603744516"
server_key_iv: "fae5bfe115235b065789db91a3b75cd1"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.452450 5734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "3da81647a548431aba7dc87cd9762b67"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "e59d7e9ef81c6694d4b3c9b603744516"
server_key_iv: "fae5bfe115235b065789db91a3b75cd1"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.456171 5734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:44.458715 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:44.459906 5734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:41:44.460038 5734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "3da81647a548431aba7dc87cd9762b67"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "e59d7e9ef81c6694d4b3c9b603744516"
server_key_iv: "fae5bfe115235b065789db91a3b75cd1"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.460145 5734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:44.474408 5734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:44.475031 5734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:44.475210 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:44.481956 5734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37051
I20260927 00:41:44.481966 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37051 every 8 connection(s)
I20260927 00:41:44.482873 5734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:44.485961 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:44.490420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5734
I20260927 00:41:44.490567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:44.490823 392 external_mini_cluster.cc:1428] Setting key cfb754b4d2364cbefe99e39c295e6f3c
I20260927 00:41:44.491299 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67: Bootstrap starting.
I20260927 00:41:44.493515 5801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:44.494323 5801 log.cc:824] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:44.496282 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67: No bootstrap required, opened a new log
I20260927 00:41:44.499773 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.492189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.0.98.62:37051)
0927 00:41:44.492768 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.492782 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.492814 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:41:44.493425 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.493441 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.493858 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.494120 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:44.494492 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.495445 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.496631 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.497134 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.497253 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.498059 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:44.498063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:44.498075 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:44.498490 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:44.498542 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:44.498561 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:44.498634 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:44.498798 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:41:44.499056 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
I20260927 00:41:44.500275 5801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } }
I20260927 00:41:44.500473 5801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:44.500540 5801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da81647a548431aba7dc87cd9762b67, State: Initialized, Role: FOLLOWER
I20260927 00:41:44.501057 5801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } }
I20260927 00:41:44.501209 5801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:44.501271 5801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:44.501379 5801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:44.502404 5801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } }
I20260927 00:41:44.502599 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:44.502758 5801 leader_election.cc:302] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da81647a548431aba7dc87cd9762b67; no voters:
I20260927 00:41:44.503048 5801 leader_election.cc:288] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:44.503139 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:44.503365 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [term 1 LEADER]: Becoming Leader. State: Replica: 3da81647a548431aba7dc87cd9762b67, State: Running, Role: LEADER
I20260927 00:41:44.503655 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } }
I20260927 00:41:44.504276 5801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:44.504781 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da81647a548431aba7dc87cd9762b67. Latest consensus state: current_term: 1 leader_uuid: "3da81647a548431aba7dc87cd9762b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } } }
I20260927 00:41:44.504927 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.505177 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da81647a548431aba7dc87cd9762b67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da81647a548431aba7dc87cd9762b67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37051 } } }
I20260927 00:41:44.505251 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.505319 5811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:44.508248 5811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:44.513665 5811 catalog_manager.cc:1347] Generated new cluster ID: 56e91195ee074e879598cff0017c248a
I20260927 00:41:44.513727 5811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:44.522724 5811 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:44.523806 5811 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:44.532186 5811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da81647a548431aba7dc87cd9762b67: Generated new TSK 0
I20260927 00:41:44.532874 5811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:44.549723 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.544936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53002 (local address 127.0.98.62:37051)
0927 00:41:44.545077 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.545082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.545332 (+ 250us) server_negotiation.cc:407] Connection header received
0927 00:41:44.545462 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.545464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.545509 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.545595 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:41:44.545837 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.546362 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.547581 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.547781 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.547872 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.548164 (+ 292us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:41:44.549337 (+ 1173us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:41:44.549576 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 00:41:44.549916 5830 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:44.544845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37051 (local address 127.0.0.1:53002)
0927 00:41:44.545215 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.545230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:44.545355 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:44.545614 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:44.545617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:44.545626 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:41:44.545737 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.545743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.546464 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.546468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.547467 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.547476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.547887 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.547907 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.547993 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.548019 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:41:44.549502 (+ 1483us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 00:41:44.549634 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":168,"threads_started":1}
W20260927 00:41:44.549793 5804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53002: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42623/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 00:41:44.550503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5734
E20260927 00:41:44.562554 5831 webserver.cc:533] Webserver: sslize failed:
E20260927 00:41:44.562618 5831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T00:41:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (285 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T00:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:44Z Disabled control of system clock
I20260927 00:41:44.598719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38757
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38757
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/testchainca.cert with env {}
W20260927 00:41:44.708868 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:44.709115 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:44.709177 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:44.713150 5835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:44.713196 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:44.713212 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:44.713260 5835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:44.713297 5835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:41:44.713330 5835 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 00:41:44.713356 5835 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 00:41:44.718884 5835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:44.719875 5835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:44.720853 5835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:44.728873 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:44.729046 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:44.729215 5835 server_base.cc:1034] running on GCE node
W20260927 00:41:44.728843 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:44.730172 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:44.731196 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:44.732378 5835 hybrid_clock.cc:648] HybridClock initialized: now 1790469704732357 us; error 33 us; skew 500 ppm
I20260927 00:41:44.734229 5835 webserver.cc:460] Webserver started at http://127.0.98.62:40415/ using document root <none> and password file <none>
I20260927 00:41:44.734789 5835 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:44.734875 5835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:44.735091 5835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:44.737102 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "0d0c7d29f547436db6c6d6ba438adab7"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "2aa61c90698ff6701449ccfb54b87eb8"
server_key_iv: "d2577d573379fbc407e6e49bd575a535"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.737653 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d0c7d29f547436db6c6d6ba438adab7"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "2aa61c90698ff6701449ccfb54b87eb8"
server_key_iv: "d2577d573379fbc407e6e49bd575a535"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.741094 5835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:44.743398 5849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:44.744532 5835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:44.744658 5835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "0d0c7d29f547436db6c6d6ba438adab7"
format_stamp: "Formatted at 2026-09-27 00:41:44 on dist-test-slave-pkws"
server_key: "2aa61c90698ff6701449ccfb54b87eb8"
server_key_iv: "d2577d573379fbc407e6e49bd575a535"
server_key_version: "encryptionkey@0"
I20260927 00:41:44.744758 5835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:44.765143 5835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:44.765775 5835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:44.765966 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:44.772557 5835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38757
I20260927 00:41:44.772562 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38757 every 8 connection(s)
I20260927 00:41:44.773492 5835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:44.774015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5835
I20260927 00:41:44.774137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:44.774412 392 external_mini_cluster.cc:1428] Setting key 008c36ba43a5dc5a3e63e6d17e925492
I20260927 00:41:44.776759 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:44.782315 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7: Bootstrap starting.
I20260927 00:41:44.783301 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.775651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.98.62:38757)
0927 00:41:44.776308 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.776327 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.776391 (+ 64us) server_negotiation.cc:407] Connection header received
0927 00:41:44.777140 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.777157 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.777548 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.777876 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:44.778290 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.779242 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.780119 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.780574 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.780745 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.781762 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:44.781766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:44.781777 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:41:44.782144 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:44.782197 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:41:44.782216 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:44.782320 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:44.782434 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 00:41:44.782661 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
I20260927 00:41:44.784722 5902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:44.785593 5902 log.cc:824] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:44.786123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:41:44.787554 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7: No bootstrap required, opened a new log
I20260927 00:41:44.790769 5902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260927 00:41:44.790969 5902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:44.791045 5902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d0c7d29f547436db6c6d6ba438adab7, State: Initialized, Role: FOLLOWER
I20260927 00:41:44.791494 5902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260927 00:41:44.791631 5902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:44.791714 5902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:44.791822 5902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:44.792806 5902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260927 00:41:44.793155 5902 leader_election.cc:302] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d0c7d29f547436db6c6d6ba438adab7; no voters:
I20260927 00:41:44.793432 5902 leader_election.cc:288] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:44.793561 5907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:44.793813 5907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [term 1 LEADER]: Becoming Leader. State: Replica: 0d0c7d29f547436db6c6d6ba438adab7, State: Running, Role: LEADER
I20260927 00:41:44.794111 5907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } }
I20260927 00:41:44.794544 5902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:44.795605 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d0c7d29f547436db6c6d6ba438adab7. Latest consensus state: current_term: 1 leader_uuid: "0d0c7d29f547436db6c6d6ba438adab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } }
I20260927 00:41:44.795722 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.796048 5908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d0c7d29f547436db6c6d6ba438adab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d0c7d29f547436db6c6d6ba438adab7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38757 } } }
I20260927 00:41:44.796142 5908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:44.796219 5916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:44.799331 5916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:44.804775 5916 catalog_manager.cc:1347] Generated new cluster ID: 8781f40d71bc4df48c0c25877919ea50
I20260927 00:41:44.804839 5916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:44.814270 5916 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:44.815513 5916 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:44.821862 5916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d0c7d29f547436db6c6d6ba438adab7: Generated new TSK 0
I20260927 00:41:44.822587 5916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:44.874464 5905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:44.832694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.98.62:38757)
0927 00:41:44.832829 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:44.832833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:44.833081 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:41:44.833224 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:44.833226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:44.833271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:44.833356 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 00:41:44.833591 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.834071 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.835271 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.835468 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.835538 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.835841 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 00:41:44.874063 (+ 38222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:41:44.874347 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:41:44.874569 5905 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 00:41:44.874739 5931 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:44.832662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38757 (local address 127.0.0.1:46432)
0927 00:41:44.832987 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:44.833003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:44.833115 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:44.833374 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:44.833377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:44.833384 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 00:41:44.833495 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.833501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.834197 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.834200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.835166 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:44.835175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:44.835553 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:44.835575 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:44.835674 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:44.835702 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 00:41:44.874316 (+ 38614us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 00:41:44.874553 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260927 00:41:44.875314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5835
2026-09-27T00:41:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (303 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:45 dist-test-slave-pkws krb5kdc[5943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:45 dist-test-slave-pkws krb5kdc[5943](info): set up 2 sockets
Sep 27 00:41:45 dist-test-slave-pkws krb5kdc[5943](info): commencing operation
krb5kdc: starting...
W20260927 00:41:46.169183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:46.285235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:46.391961 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:46.392216 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:46.392256 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:46.396180 5956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:46.396237 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:46.396256 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:46.396273 5956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:46.396288 5956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:46.401620 5956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:46.402608 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:46.403574 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:46.411810 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:46.411923 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:46.411936 5956 server_base.cc:1034] running on GCE node
W20260927 00:41:46.411810 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:46.412565 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:46.413693 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:46.414876 5956 hybrid_clock.cc:648] HybridClock initialized: now 1790469706414860 us; error 29 us; skew 500 ppm
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:46.423381 5956 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:46.424322 5956 webserver.cc:460] Webserver started at http://127.0.98.62:37269/ using document root <none> and password file <none>
I20260927 00:41:46.424883 5956 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:46.425004 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:46.425235 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:46.427259 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "28173441f665402ba4f0452d50206f0e"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "5fbcb21116fb043496563cf0bdcab513"
server_key_iv: "c219c2fe0ba0ff729e558fd60d800818"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.427819 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "28173441f665402ba4f0452d50206f0e"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "5fbcb21116fb043496563cf0bdcab513"
server_key_iv: "c219c2fe0ba0ff729e558fd60d800818"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.431511 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:41:46.433984 5971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.435664 5956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:46.435791 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "28173441f665402ba4f0452d50206f0e"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "5fbcb21116fb043496563cf0bdcab513"
server_key_iv: "c219c2fe0ba0ff729e558fd60d800818"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.435905 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:46.459230 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:46.462335 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:46.462507 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:46.469525 5956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38637
I20260927 00:41:46.469789 6023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38637 every 8 connection(s)
I20260927 00:41:46.470521 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:46.470727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 5956
I20260927 00:41:46.470861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:46.471220 392 external_mini_cluster.cc:1428] Setting key 7596983b3cd12e1ebc7c16da97e09f39
I20260927 00:41:46.473757 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:46.480299 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e: Bootstrap starting.
I20260927 00:41:46.482415 6024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:46.483172 6024 log.cc:824] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:46.485059 6024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e: No bootstrap required, opened a new log
I20260927 00:41:46.485314 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.472609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53904 (local address 127.0.98.62:38637)
0927 00:41:46.473016 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:46.473036 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:46.473083 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:41:46.473713 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:46.473728 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:46.474091 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:46.474363 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:46.475263 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.476249 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.477432 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.477882 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.478011 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:46.480728 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:46.480733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:46.480745 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:46.480770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:46.482585 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.483036 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.483041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.483045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.483149 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.483431 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.483434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.483436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.483789 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:46.483994 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:46.484259 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 00:41:46.484530 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":151,"threads_started":1}
I20260927 00:41:46.488574 6024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } }
I20260927 00:41:46.488778 6024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:46.488852 6024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28173441f665402ba4f0452d50206f0e, State: Initialized, Role: FOLLOWER
I20260927 00:41:46.489322 6024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } }
I20260927 00:41:46.489464 6024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:46.489548 6024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:46.489660 6024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:46.490556 6024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } }
I20260927 00:41:46.490898 6024 leader_election.cc:302] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28173441f665402ba4f0452d50206f0e; no voters:
I20260927 00:41:46.491182 6024 leader_election.cc:288] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:46.491328 6029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:46.491605 6029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [term 1 LEADER]: Becoming Leader. State: Replica: 28173441f665402ba4f0452d50206f0e, State: Running, Role: LEADER
I20260927 00:41:46.491918 6029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } }
I20260927 00:41:46.492395 6024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:46.493537 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28173441f665402ba4f0452d50206f0e. Latest consensus state: current_term: 1 leader_uuid: "28173441f665402ba4f0452d50206f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } } }
I20260927 00:41:46.493630 6030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28173441f665402ba4f0452d50206f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28173441f665402ba4f0452d50206f0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38637 } } }
I20260927 00:41:46.493674 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:46.493697 6030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:46.494244 6038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:46.496807 6038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:46.501978 6038 catalog_manager.cc:1347] Generated new cluster ID: 1aa56644f6994ae7916621e8aa9ef1c1
I20260927 00:41:46.502038 6038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:46.510502 6038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:46.511646 6038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:46.519381 6038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28173441f665402ba4f0452d50206f0e: Generated new TSK 0
I20260927 00:41:46.520135 6038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:46.545938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:46.652987 6050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:46.653198 6050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:46.653270 6050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:46.657236 6050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:46.657294 6050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:46.657416 6050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:46.662514 6050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:46.663434 6050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:46.664382 6050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:46.673768 6056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:46.673748 6055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:46.673750 6058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:46.674029 6050 server_base.cc:1034] running on GCE node
I20260927 00:41:46.674408 6050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:46.674993 6050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:46.676169 6050 hybrid_clock.cc:648] HybridClock initialized: now 1790469706676147 us; error 47 us; skew 500 ppm
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:46.678871 6050 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:41:46.679634 6050 webserver.cc:460] Webserver started at http://127.0.98.1:43747/ using document root <none> and password file <none>
I20260927 00:41:46.680158 6050 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:46.680244 6050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:46.680454 6050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:46.682457 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f9a2b1b588d457ead8291f7bb6ca265"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "7b6798eae9ef21cb5f53104d56eacfb9"
server_key_iv: "f5896c133058bd769bca21eb0165a33c"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.683045 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f9a2b1b588d457ead8291f7bb6ca265"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "7b6798eae9ef21cb5f53104d56eacfb9"
server_key_iv: "f5896c133058bd769bca21eb0165a33c"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.686622 6050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 00:41:46.689025 6065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.690445 6050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 00:41:46.690534 6050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "8f9a2b1b588d457ead8291f7bb6ca265"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "7b6798eae9ef21cb5f53104d56eacfb9"
server_key_iv: "f5896c133058bd769bca21eb0165a33c"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.690610 6050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:46.718281 6050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:46.721554 6050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:46.721709 6050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:46.722231 6050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:46.722968 6050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:46.723021 6050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.723067 6050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:46.723083 6050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.731818 6050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36973
I20260927 00:41:46.731844 6178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36973 every 8 connection(s)
I20260927 00:41:46.732642 6050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:46.741674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6050
I20260927 00:41:46.741806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:46.742059 392 external_mini_cluster.cc:1428] Setting key 514db2c0c3c50be175793a677cc0e593
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:46.746476 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.734502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49033 (local address 127.0.98.62:38637)
0927 00:41:46.734668 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:46.734673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:46.735494 (+ 821us) server_negotiation.cc:407] Connection header received
0927 00:41:46.736800 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:46.736803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:46.736850 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:46.736955 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:46.738083 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.738624 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.739746 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.739998 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.740088 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:46.742722 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:46.742726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:46.742729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:46.742754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:46.744232 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.744837 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.744840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.744841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.744981 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.745372 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.745375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.745377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.745589 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:46.745699 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:46.746189 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 00:41:46.746306 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:41:46.747303 6181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.734908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38637 (local address 127.0.98.1:49033)
0927 00:41:46.735335 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:46.735376 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:46.736598 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:46.737128 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:46.737135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:46.737476 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:46.737928 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:46.737940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.738729 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.738731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:46.739628 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:46.739637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.740142 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.740144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:46.740280 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:46.740869 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:46.740884 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:46.742563 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:46.744344 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:46.744353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:46.744370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:46.744733 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:46.745097 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:46.745100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:46.745102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:46.745266 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:46.745697 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:46.745703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:46.745929 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:41:46.746142 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 00:41:46.746458 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260927 00:41:46.748495 6179 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38637
I20260927 00:41:46.748723 6179 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:46.749233 6179 heartbeater.cc:511] Master 127.0.98.62:38637 requested a full tablet report, sending...
I20260927 00:41:46.750649 5988 ts_manager.cc:194] Registered new tserver with Master: 8f9a2b1b588d457ead8291f7bb6ca265 (127.0.98.1:36973)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:41:46.751770 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:41:46.767491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:46.878098 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:46.878336 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:46.878378 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:46.882395 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:46.882441 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:46.882496 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:46.887660 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:46.888562 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:46.889555 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:46.899055 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:46.899093 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:46.899233 6184 server_base.cc:1034] running on GCE node
W20260927 00:41:46.899049 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:46.899983 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:46.900524 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:46.901679 6184 hybrid_clock.cc:648] HybridClock initialized: now 1790469706901673 us; error 38 us; skew 500 ppm
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:46.904390 6184 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:41:46.905293 6184 webserver.cc:460] Webserver started at http://127.0.98.2:37955/ using document root <none> and password file <none>
I20260927 00:41:46.905819 6184 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:46.905877 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:46.906112 6184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:46.908083 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "87c18056cb91419a88592c0a6414b706"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "9ea0b872fd8a06076fd63b9302d5d071"
server_key_iv: "a1a683256a41fbdec7ac5d5a25ab8ccd"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.908614 6184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "87c18056cb91419a88592c0a6414b706"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "9ea0b872fd8a06076fd63b9302d5d071"
server_key_iv: "a1a683256a41fbdec7ac5d5a25ab8ccd"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.912142 6184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:46.914644 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.915834 6184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:46.915969 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "87c18056cb91419a88592c0a6414b706"
format_stamp: "Formatted at 2026-09-27 00:41:46 on dist-test-slave-pkws"
server_key: "9ea0b872fd8a06076fd63b9302d5d071"
server_key_iv: "a1a683256a41fbdec7ac5d5a25ab8ccd"
server_key_version: "encryptionkey@0"
I20260927 00:41:46.916091 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:46.934537 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:46.937309 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:46.937500 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:46.938036 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:46.938832 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:46.938900 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.938968 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:46.939020 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:46.947701 6184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42759
I20260927 00:41:46.947726 6312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42759 every 8 connection(s)
I20260927 00:41:46.948544 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:46.953274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6184
I20260927 00:41:46.953394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:46.953689 392 external_mini_cluster.cc:1428] Setting key b48a9258d7a02c2d45fc11b928fffa5b
Sep 27 00:41:46 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:46.962978 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.950199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49365 (local address 127.0.98.62:38637)
0927 00:41:46.950336 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:46.950340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:46.951118 (+ 778us) server_negotiation.cc:407] Connection header received
0927 00:41:46.952461 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:46.952465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:46.952519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:46.952628 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:46.953880 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.954585 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.956297 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.956539 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.956653 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:46.959190 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:46.959195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:46.959201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:46.959233 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:46.960979 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.961447 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.961452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.961456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.961558 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:46.961865 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:46.961869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:46.961872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:46.962111 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:46.962218 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:46.962648 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 00:41:46.962777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:46.963625 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:46.950554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38637 (local address 127.0.98.2:49365)
0927 00:41:46.950981 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:46.951020 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:46.952251 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:46.952790 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:46.952796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:46.953191 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:46.953716 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:46.953730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.954705 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.954708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:46.956169 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:46.956184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:46.956686 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:46.956694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:46.956971 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:46.957604 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:46.957630 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:46.958997 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:46.961090 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:46.961094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:46.961104 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:46.961333 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:46.961654 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:46.961656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:46.961657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:46.961773 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:46.962221 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:46.962225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:46.962422 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 00:41:46.962563 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:41:46.962803 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260927 00:41:46.964763 6313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38637
I20260927 00:41:46.965040 6313 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:46.965531 6313 heartbeater.cc:511] Master 127.0.98.62:38637 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:46.966638 5988 ts_manager.cc:194] Registered new tserver with Master: 87c18056cb91419a88592c0a6414b706 (127.0.98.2:42759)
I20260927 00:41:46.967417 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:41:46.980254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:47.090842 6318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:47.091120 6318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:47.091194 6318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:47.096128 6318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:47.096208 6318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:47.096294 6318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:47.101955 6318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:47.102974 6318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:47.103988 6318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:47.113443 6323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:47.113492 6318 server_base.cc:1034] running on GCE node
W20260927 00:41:47.113404 6324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:47.113406 6326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:47.114130 6318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:47.114761 6318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:47.115945 6318 hybrid_clock.cc:648] HybridClock initialized: now 1790469707115911 us; error 35 us; skew 500 ppm
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.118728 6318 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:41:47.119652 6318 webserver.cc:460] Webserver started at http://127.0.98.3:38299/ using document root <none> and password file <none>
I20260927 00:41:47.120229 6318 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:47.120322 6318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:47.120551 6318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:47.122700 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bb768a8fb7a4ff58621e7bffce5e724"
format_stamp: "Formatted at 2026-09-27 00:41:47 on dist-test-slave-pkws"
server_key: "4cc00b0ab93ade39ed6737adf6c13e50"
server_key_iv: "baf73c5cf3cbbd7c1f0517cb13869f4b"
server_key_version: "encryptionkey@0"
I20260927 00:41:47.123302 6318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bb768a8fb7a4ff58621e7bffce5e724"
format_stamp: "Formatted at 2026-09-27 00:41:47 on dist-test-slave-pkws"
server_key: "4cc00b0ab93ade39ed6737adf6c13e50"
server_key_iv: "baf73c5cf3cbbd7c1f0517cb13869f4b"
server_key_version: "encryptionkey@0"
I20260927 00:41:47.127040 6318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:47.129403 6333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:47.130610 6318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:47.130743 6318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "8bb768a8fb7a4ff58621e7bffce5e724"
format_stamp: "Formatted at 2026-09-27 00:41:47 on dist-test-slave-pkws"
server_key: "4cc00b0ab93ade39ed6737adf6c13e50"
server_key_iv: "baf73c5cf3cbbd7c1f0517cb13869f4b"
server_key_version: "encryptionkey@0"
I20260927 00:41:47.130852 6318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:47.146539 6318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:47.149359 6318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:47.149583 6318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:47.150178 6318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:47.151018 6318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:47.151094 6318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:47.151170 6318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:47.151224 6318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:47.160437 6318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34371
I20260927 00:41:47.160483 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34371 every 8 connection(s)
I20260927 00:41:47.161339 6318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:47.165843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6318
I20260927 00:41:47.165966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:47.166234 392 external_mini_cluster.cc:1428] Setting key 66ea21209310f413c74d1d87dceb147a
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:47.174732 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.163016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58965 (local address 127.0.98.62:38637)
0927 00:41:47.163156 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.163160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.164012 (+ 852us) server_negotiation.cc:407] Connection header received
0927 00:41:47.165435 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.165439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.165497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.165599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.166919 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.167605 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.168713 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.169016 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.169126 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:47.171428 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:47.171432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:47.171435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:47.171461 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:47.172739 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.173271 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.173276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.173278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.173364 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.173728 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.173732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.173734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.173909 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:47.174023 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:47.174460 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 00:41:47.174576 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:41:47.175551 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.163368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38637 (local address 127.0.98.3:58965)
0927 00:41:47.163810 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:47.163848 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:47.165224 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:47.165778 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:47.165785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:47.166208 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:47.166786 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:47.166801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.167724 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.167727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:47.168584 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:47.168593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.169157 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.169161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:47.169325 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:47.169831 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:47.169847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:47.171277 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:47.172855 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:47.172860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:47.172881 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:47.173164 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:47.173463 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:47.173466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:47.173468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:47.173628 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:47.174019 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:47.174025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:47.174241 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 00:41:47.174420 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 00:41:47.174716 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260927 00:41:47.176600 6447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38637
I20260927 00:41:47.176832 6447 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:47.177297 6447 heartbeater.cc:511] Master 127.0.98.62:38637 requested a full tablet report, sending...
I20260927 00:41:47.178318 5988 ts_manager.cc:194] Registered new tserver with Master: 8bb768a8fb7a4ff58621e7bffce5e724 (127.0.98.3:34371)
I20260927 00:41:47.179039 5988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58965
I20260927 00:41:47.179718 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/test-admin.keytab.
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.194868 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 00:41:47.197494 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:47.196099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.0.98.1:36973)
0927 00:41:47.196400 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.196413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.196435 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:41:47.196480 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.196485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.196677 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.196792 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.197321 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35618: BlockingRecv error: recv got EOF from 127.0.0.1:35618 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":128,"threads_started":1}
W20260927 00:41:47.197535 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.198709 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.199636 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:47.209115 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.200418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.0.98.1:36973)
0927 00:41:47.200567 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.200570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.200584 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:47.200677 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.200679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.200715 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.200781 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.201375 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.202266 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.203062 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.203471 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.203568 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:47.205750 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:47.205758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:47.205771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:47.205792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:47.207459 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.207820 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.207825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.207826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.207898 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.208187 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.208189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.208190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.208404 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:47.208544 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:47.208750 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:41:47.208921 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:41:47.212605 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:47.211770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.0.98.1:36973)
0927 00:41:47.211857 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.211860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.212006 (+ 146us) server_negotiation.cc:407] Connection header received
0927 00:41:47.212072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.212074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.212106 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.212166 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.212516 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35632: BlockingRecv error: recv got EOF from 127.0.0.1:35632 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 00:41:47.212949 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.213948 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.214839 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:47.223269 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.215567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.98.1:36973)
0927 00:41:47.215675 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.215678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.215811 (+ 133us) server_negotiation.cc:407] Connection header received
0927 00:41:47.215948 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.215951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.215996 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.216074 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.216575 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.217146 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.217917 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.218144 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.218252 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:47.220434 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:47.220438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:47.220443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:47.220468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:47.221694 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.222164 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.222167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.222168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.222237 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.222532 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.222535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.222537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.222686 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:47.222812 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:47.223041 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 00:41:47.223147 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:41:47.225790 6454 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:47.224826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.0.98.1:36973)
0927 00:41:47.224976 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.224982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.225033 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:41:47.225189 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.225192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.225232 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.225309 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.225701 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35648: BlockingRecv error: recv got EOF from 127.0.0.1:35648 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 00:41:47.225919 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790469678562534-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.226840 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:47.227718 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[5943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:41:47.235301 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:47.228444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.0.98.1:36973)
0927 00:41:47.228568 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:47.228571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:47.228605 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:41:47.228692 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:47.228693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:47.228722 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:47.228784 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:47.229275 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.229746 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.230447 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:47.230667 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:47.230741 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:47.232725 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:47.232727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:47.232729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:47.232748 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:47.233971 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.234326 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.234329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.234330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.234395 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:47.234687 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:47.234690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:47.234691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:47.234832 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:47.234925 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:47.235131 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 00:41:47.235211 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 00:41:47.236181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6050
I20260927 00:41:47.242262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6184
I20260927 00:41:47.247315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6318
I20260927 00:41:47.252467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 5956
2026-09-27T00:41:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2132 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[6467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[6467](info): set up 2 sockets
Sep 27 00:41:47 dist-test-slave-pkws krb5kdc[6467](info): commencing operation
krb5kdc: starting...
W20260927 00:41:48.309947 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:48 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:41:48.439579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:48.545536 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:48.545770 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:48.545807 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:48.550042 6480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:48.550100 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:48.550124 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:48.550141 6480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:48.550158 6480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:48.555603 6480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:48.556679 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:48.557703 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:48.566092 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:48.566105 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:48.566174 6480 server_base.cc:1034] running on GCE node
W20260927 00:41:48.566136 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:48.566931 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:48.567931 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:48.569083 6480 hybrid_clock.cc:648] HybridClock initialized: now 1790469708569069 us; error 28 us; skew 500 ppm
Sep 27 00:41:48 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:48.571813 6480 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:41:48.572664 6480 webserver.cc:460] Webserver started at http://127.0.98.62:43799/ using document root <none> and password file <none>
I20260927 00:41:48.573266 6480 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:48.573351 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:48.573577 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:48.575685 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "f680812cde3b440791bed40f5e7a9744"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "465648663b2818570151a08779e6cf9f"
server_key_iv: "eccccd6149be28a4d4804d951d1f8f66"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.576257 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "f680812cde3b440791bed40f5e7a9744"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "465648663b2818570151a08779e6cf9f"
server_key_iv: "eccccd6149be28a4d4804d951d1f8f66"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.579903 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:48.582311 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:48.583454 6480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:48.583602 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "f680812cde3b440791bed40f5e7a9744"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "465648663b2818570151a08779e6cf9f"
server_key_iv: "eccccd6149be28a4d4804d951d1f8f66"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.583707 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:48.599999 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:48.602891 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:48.603101 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:48.610046 6480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37431
I20260927 00:41:48.610349 6547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37431 every 8 connection(s)
I20260927 00:41:48.610967 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:48.613638 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:48.615178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6480
I20260927 00:41:48.615320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:48.615614 392 external_mini_cluster.cc:1428] Setting key 6c7c624c1102327d2b7b8aad53cce5b5
I20260927 00:41:48.620056 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Bootstrap starting.
I20260927 00:41:48.622560 6548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:48.623404 6548 log.cc:824] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:41:48 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:48.625725 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: No bootstrap required, opened a new log
I20260927 00:41:48.629733 6548 raft_consensus.cc:348] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 00:41:48.629922 6548 raft_consensus.cc:374] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:48.629956 6548 raft_consensus.cc:729] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680812cde3b440791bed40f5e7a9744, State: Initialized, Role: FOLLOWER
I20260927 00:41:48.630224 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.616997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.0.98.62:37431)
0927 00:41:48.617382 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:48.617400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:48.617442 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:41:48.618264 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:48.618287 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:48.618673 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:48.619034 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:48.619674 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.621068 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.621893 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.622560 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.622717 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:48.625720 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:48.625726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:48.625740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:48.625785 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:48.627385 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:48.627940 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:48.627945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:48.627949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:48.628049 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:48.628316 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:48.628318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:48.628320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:48.628657 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:48.628809 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:48.629142 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 00:41:48.629445 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260927 00:41:48.630414 6548 consensus_queue.cc:260] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 00:41:48.630549 6548 raft_consensus.cc:388] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:48.630597 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:48.630681 6548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:48.631532 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 00:41:48.631855 6548 leader_election.cc:302] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f680812cde3b440791bed40f5e7a9744; no voters:
I20260927 00:41:48.632171 6548 leader_election.cc:288] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:48.632284 6553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:48.632469 6553 raft_consensus.cc:686] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 LEADER]: Becoming Leader. State: Replica: f680812cde3b440791bed40f5e7a9744, State: Running, Role: LEADER
I20260927 00:41:48.632754 6553 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } }
I20260927 00:41:48.633312 6548 sys_catalog.cc:564] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:48.634423 6554 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260927 00:41:48.634605 6554 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:48.634449 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f680812cde3b440791bed40f5e7a9744. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } }
I20260927 00:41:48.634876 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:48.635520 6562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:48.638206 6562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:48.643239 6562 catalog_manager.cc:1347] Generated new cluster ID: 8a74c60375024ff18669926adc5eea10
I20260927 00:41:48.643293 6562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:48.655503 6562 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:48.656682 6562 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:48.666699 6562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Generated new TSK 0
I20260927 00:41:48.667312 6562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:41:48.697491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:48.805063 6574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:48.805323 6574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:48.805366 6574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:48.809319 6574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:48.809372 6574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:48.809427 6574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:48.814493 6574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:48.815375 6574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:48.816282 6574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:48.825337 6580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:48.825340 6582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:48.825461 6574 server_base.cc:1034] running on GCE node
W20260927 00:41:48.825363 6579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:48.825949 6574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:48.826570 6574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:48.827750 6574 hybrid_clock.cc:648] HybridClock initialized: now 1790469708827737 us; error 26 us; skew 500 ppm
Sep 27 00:41:48 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:48.830394 6574 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:41:48.831210 6574 webserver.cc:460] Webserver started at http://127.0.98.1:39463/ using document root <none> and password file <none>
I20260927 00:41:48.831740 6574 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:48.831832 6574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:48.832048 6574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:48.834187 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b58ea492466483d909e1674d38a9a2d"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "5ac121a3a6b178515282a9238dd0d841"
server_key_iv: "bcd1fd7bcc867b47f4a3a30f17dbdbbc"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.834806 6574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b58ea492466483d909e1674d38a9a2d"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "5ac121a3a6b178515282a9238dd0d841"
server_key_iv: "bcd1fd7bcc867b47f4a3a30f17dbdbbc"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.838369 6574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:41:48.840619 6589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:48.841866 6574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:48.842010 6574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "0b58ea492466483d909e1674d38a9a2d"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "5ac121a3a6b178515282a9238dd0d841"
server_key_iv: "bcd1fd7bcc867b47f4a3a30f17dbdbbc"
server_key_version: "encryptionkey@0"
I20260927 00:41:48.842116 6574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:48.857187 6574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:48.860352 6574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:48.860556 6574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:48.861215 6574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:48.862017 6574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:48.862087 6574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:48.862157 6574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:48.862217 6574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:48.870852 6574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34025
I20260927 00:41:48.870877 6702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34025 every 8 connection(s)
I20260927 00:41:48.871659 6574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:48.873060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6574
I20260927 00:41:48.873189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:48.873533 392 external_mini_cluster.cc:1428] Setting key 70eb0b898c9b527b78a88309a7faf26b
Sep 27 00:41:48 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 00:41:48.886171 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.873500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50597 (local address 127.0.98.62:37431)
0927 00:41:48.873714 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:48.873718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:48.875045 (+ 1327us) server_negotiation.cc:407] Connection header received
0927 00:41:48.876103 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:48.876109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:48.876173 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:48.876275 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:48.877521 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.878061 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.879270 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.879501 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.879614 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:48.882426 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:48.882429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:48.882431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:48.882453 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:48.883941 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:48.884507 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:48.884513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:48.884516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:48.884676 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:48.885111 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:48.885114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:48.885116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:48.885300 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:48.885408 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:48.885889 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 00:41:48.886004 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 00:41:48.887035 6705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:48.873976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:50597)
0927 00:41:48.874876 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:48.874940 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:48.875852 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:48.876452 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:48.876460 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:48.876882 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:48.877392 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:48.877405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.878207 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.878216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:48.879146 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:48.879156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:48.879647 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:48.879651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:48.879811 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:48.880382 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:48.880397 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:48.882176 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:48.884112 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:48.884117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:48.884133 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:48.884400 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:48.884776 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:48.884780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:48.884782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:48.884995 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:48.885406 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:48.885412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:48.885653 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 00:41:48.885835 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 00:41:48.886147 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":548,"threads_started":1}
I20260927 00:41:48.888286 6703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:48.888545 6703 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:48.889089 6703 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:48.890614 6512 ts_manager.cc:194] Registered new tserver with Master: 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025)
I20260927 00:41:48.891706 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:41:48.900460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:49.007853 6708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:49.008126 6708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:49.008216 6708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:49.012188 6708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:49.012266 6708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:49.012372 6708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:49.017395 6708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:49.018361 6708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:49.019318 6708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:41:49.028652 6708 server_base.cc:1034] running on GCE node
W20260927 00:41:49.028610 6714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.028681 6716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.028609 6713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.029219 6708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.029794 6708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.030968 6708 hybrid_clock.cc:648] HybridClock initialized: now 1790469709030955 us; error 25 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.033636 6708 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:41:49.034538 6708 webserver.cc:460] Webserver started at http://127.0.98.2:43123/ using document root <none> and password file <none>
I20260927 00:41:49.035012 6708 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.035100 6708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.035295 6708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:49.037192 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "40e44f10c2ec43a792344daf15ca4d52"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "2fd2775f585914ae8e2a7b855b1f05c7"
server_key_iv: "ea2e9f947fc91ea6f72edbdd4f4f3770"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.037676 6708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40e44f10c2ec43a792344daf15ca4d52"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "2fd2775f585914ae8e2a7b855b1f05c7"
server_key_iv: "ea2e9f947fc91ea6f72edbdd4f4f3770"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.041167 6708 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:49.043335 6723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.044458 6708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:49.044571 6708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "40e44f10c2ec43a792344daf15ca4d52"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "2fd2775f585914ae8e2a7b855b1f05c7"
server_key_iv: "ea2e9f947fc91ea6f72edbdd4f4f3770"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.044672 6708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.056528 6708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:49.059139 6708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:49.059319 6708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:49.059839 6708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:49.060582 6708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:49.060627 6708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.060706 6708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:49.060750 6708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.069617 6708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41039
I20260927 00:41:49.069636 6836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41039 every 8 connection(s)
I20260927 00:41:49.070449 6708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:49.076364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6708
I20260927 00:41:49.076493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:49.076789 392 external_mini_cluster.cc:1428] Setting key 05f85d7572733e84a40051af71352fed
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:49.085500 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.072170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51441 (local address 127.0.98.62:37431)
0927 00:41:49.072298 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.072301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.073132 (+ 831us) server_negotiation.cc:407] Connection header received
0927 00:41:49.074483 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.074487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.074544 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.074645 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.075753 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.076397 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.077827 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.078188 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.078280 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.081478 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.081482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.081484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.081517 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.083275 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.083812 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.083819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.083822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.083922 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.084261 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.084266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.084271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.084538 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.084650 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.085182 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.085314 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:41:49.086194 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.072529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:51441)
0927 00:41:49.072985 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.073035 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.074268 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.074816 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.074822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.075159 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.075613 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.075625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.076602 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.076611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.077702 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.077712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.078349 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.078354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.078556 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.079141 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.079166 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.081290 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.083389 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.083393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.083403 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.083687 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.084019 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.084022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.084023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.084147 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.084648 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.084652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.084868 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.085082 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.085321 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:41:49.087405 6837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:49.087677 6837 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:49.088224 6837 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:49.089339 6512 ts_manager.cc:194] Registered new tserver with Master: 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 00:41:49.089946 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:41:49.103618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:49.212318 6842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:49.212545 6842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:49.212585 6842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:49.216444 6842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:49.216493 6842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:49.216547 6842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:49.221452 6842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:49.222345 6842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:49.223201 6842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:49.232169 6850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.232200 6848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.232167 6847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.232558 6842 server_base.cc:1034] running on GCE node
I20260927 00:41:49.232930 6842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.233484 6842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.234650 6842 hybrid_clock.cc:648] HybridClock initialized: now 1790469709234626 us; error 55 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.237403 6842 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:41:49.238138 6842 webserver.cc:460] Webserver started at http://127.0.98.3:44489/ using document root <none> and password file <none>
I20260927 00:41:49.238613 6842 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.238668 6842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.238880 6842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:49.240813 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "fb645d1788f346c85835cf4da24014f1"
server_key_iv: "5b20cd01d358b1a05c042cea1b3c4b5d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.241365 6842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "fb645d1788f346c85835cf4da24014f1"
server_key_iv: "5b20cd01d358b1a05c042cea1b3c4b5d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.244760 6842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:41:49.247159 6857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.248396 6842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:49.248507 6842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "fb645d1788f346c85835cf4da24014f1"
server_key_iv: "5b20cd01d358b1a05c042cea1b3c4b5d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.248606 6842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.266697 6842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:49.269821 6842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:49.270000 6842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:49.270543 6842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:49.271306 6842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:49.271353 6842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.271414 6842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:49.271454 6842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.280145 6842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43025
I20260927 00:41:49.280467 6970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43025 every 8 connection(s)
I20260927 00:41:49.281080 6842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:49.289573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6842
I20260927 00:41:49.289685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:49.289978 392 external_mini_cluster.cc:1428] Setting key d14e773da2d96ce2721fe567886a3edb
I20260927 00:41:49.293792 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.282669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35287 (local address 127.0.98.62:37431)
0927 00:41:49.282809 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.282813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.283564 (+ 751us) server_negotiation.cc:407] Connection header received
0927 00:41:49.284438 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.284441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.284498 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.284599 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.285745 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.286278 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.287456 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.287750 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.287861 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.290175 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.290178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.290180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.290203 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.291693 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.292189 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.292194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.292196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.292278 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.292712 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.292717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.292719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.292949 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.293079 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.293524 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.293637 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:49.294529 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.283018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:35287)
0927 00:41:49.283431 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.283470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.284221 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.284766 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.284773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.285161 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.285605 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.285617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.286404 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.286408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.287324 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.287333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.287903 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.287906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.288067 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.288528 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.288543 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.289971 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.291814 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.291820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.291833 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.292086 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.292371 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.292376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.292378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.292603 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.293073 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.293078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.293304 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.293484 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.293783 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260927 00:41:49.295554 6971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:49.295745 6971 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:49.296172 6971 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:49.297209 6512 ts_manager.cc:194] Registered new tserver with Master: 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:49.297761 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35287
I20260927 00:41:49.304092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 00:41:49.329237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35707
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431,127.0.98.61:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:49.436968 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:49.437252 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:49.437335 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:49.441414 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:49.441470 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:49.441489 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:49.441556 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:49.441586 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:49.447506 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431,127.0.98.61:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
W20260927 00:41:49.447790 6976 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37431,127.0.98.61:35707'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 00:41:49.448793 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:49.449868 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:49.458240 6984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.458271 6981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.458261 6982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.458595 6976 server_base.cc:1034] running on GCE node
I20260927 00:41:49.459156 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.460206 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.461388 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790469709461362 us; error 48 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.464090 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:41:49.465056 6976 webserver.cc:460] Webserver started at http://127.0.98.61:40465/ using document root <none> and password file <none>
I20260927 00:41:49.465591 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.465680 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.465898 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:49.467816 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/instance:
uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "f249e8134d42ce7f136c068ce1dc9416"
server_key_iv: "99ddc80c1c8a18b93be8619639e1de4d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.468362 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal/instance:
uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "f249e8134d42ce7f136c068ce1dc9416"
server_key_iv: "99ddc80c1c8a18b93be8619639e1de4d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.471853 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:41:49.474190 6991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.475384 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:49.475525 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "f249e8134d42ce7f136c068ce1dc9416"
server_key_iv: "99ddc80c1c8a18b93be8619639e1de4d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.475627 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.509868 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:49.513106 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:49.513304 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:49.520196 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35707
I20260927 00:41:49.520232 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35707 every 8 connection(s)
I20260927 00:41:49.521080 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:41:49.523650 7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:49.525079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 6976
I20260927 00:41:49.527834 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:49.539247 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.529670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.62:37431)
0927 00:41:49.529816 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.529820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.530076 (+ 256us) server_negotiation.cc:407] Connection header received
0927 00:41:49.530200 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.530203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.530267 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.530334 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.531033 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.531692 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.532852 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.533219 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.533334 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.535869 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.535872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.535875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.535906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.537417 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.537933 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.537937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.537938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.538036 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.538307 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.538309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.538310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.538494 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.538598 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.538965 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.539083 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:41:49.540096 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.526504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.98.61:35707)
0927 00:41:49.526911 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.526928 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.526969 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:41:49.527761 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.527781 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.528178 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.528523 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.529220 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.530549 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.531548 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.532200 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.532344 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.535106 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.535111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.535119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.535139 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.537084 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.537610 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.537618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.537621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.537749 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.538069 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.538072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.538075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.538497 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.538687 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.538886 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.539230 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":276,"thread_start_us":131,"threads_started":1}
I20260927 00:41:49.540096 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.529599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47330)
0927 00:41:49.529946 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.529991 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.530109 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.530396 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.530401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.530795 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.530920 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.530925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.531813 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.531816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.532735 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.532746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.533413 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.533416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.533493 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.534011 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.534026 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.535724 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.537507 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.537513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.537520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.537846 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.538139 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.538141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.538143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.538217 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.538592 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.538598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.538734 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.538887 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.539182 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":130,"threads_started":1}
I20260927 00:41:49.541421 7044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:41:49.548775 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.541871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48348)
0927 00:41:49.541981 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.541993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.542082 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.542317 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.542319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.542438 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.542516 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.542520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.543480 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.543483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.543966 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.543974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.544353 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.544355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.544423 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.544855 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.544875 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.546040 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.547380 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.547382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.547384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.547604 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.547860 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.547863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.547868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.547957 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.548332 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.548334 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.548407 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.548502 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.548606 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 00:41:49.548779 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.541981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.61:35707)
0927 00:41:49.542078 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.542082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.542094 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:49.542178 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.542181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.542220 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.542301 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.542607 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.543390 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.544064 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.544265 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.544336 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.546176 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.546178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.546180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.546199 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.547288 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.547696 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.547698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.547700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.547772 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.548033 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.548035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.548036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.548224 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.548301 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.548551 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.548656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":44}
I20260927 00:41:49.551419 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Bootstrap starting.
I20260927 00:41:49.553718 7044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:49.554430 7044 log.cc:824] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:49.556190 7044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: No bootstrap required, opened a new log
I20260927 00:41:49.559089 7044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } }
I20260927 00:41:49.559309 7044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:49.559388 7044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a5cbbba72f4a0eba57906c09cf85a4, State: Initialized, Role: FOLLOWER
I20260927 00:41:49.559806 7044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } }
I20260927 00:41:49.560252 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } } }
I20260927 00:41:49.560357 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:49.560997 7044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:41:49.565611 7061 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:49.565696 7061 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:49.567734 6976 master_runner.cc:428] Detected that this master 60a5cbbba72f4a0eba57906c09cf85a4 is joining an existing cluster
I20260927 00:41:49.567806 6976 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35707
I20260927 00:41:49.579180 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.570011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47332)
0927 00:41:49.570370 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.570385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.570497 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.570900 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.570903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.571283 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.571516 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.571524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.572624 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.572626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.573294 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.573308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.573794 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.573797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.573889 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.574398 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.574413 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.574884 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.577128 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.577131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.577134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.577358 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.577672 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.577676 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.577678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.577792 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.578641 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.578644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.578727 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.578827 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.578996 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":38,"thread_start_us":209,"threads_started":1}
I20260927 00:41:49.579198 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.570074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.62:37431)
0927 00:41:49.570203 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.570208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.570481 (+ 273us) server_negotiation.cc:407] Connection header received
0927 00:41:49.570632 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.570636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.570691 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.570769 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.571646 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.572496 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.573414 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.573639 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.573779 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.575037 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.575040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.575044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.575083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.577001 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.577472 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.577476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.577478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.577563 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.577944 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.577950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.577952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.578188 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.578289 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.578898 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.579021 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:41:49.580255 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.570099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.61:35707)
0927 00:41:49.570203 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.570208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.570641 (+ 433us) server_negotiation.cc:407] Connection header received
0927 00:41:49.570766 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.570769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.570810 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.570888 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.572636 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.573369 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.574464 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.574804 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.574922 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.576166 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.576170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.576177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.576198 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.578184 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.578648 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.578651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.578652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.578766 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.579058 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.579060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.579062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.579589 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.579663 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.579993 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.580135 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":28}
I20260927 00:41:49.580261 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.570154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48362)
0927 00:41:49.570558 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.570572 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.570670 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.570902 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.570904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.571285 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.571519 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.571524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.573485 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.573487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.574358 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.574369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.574946 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.574950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.575087 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.575552 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.575569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.576048 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.578294 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.578297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.578299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.578533 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.578872 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.578875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.578877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.578962 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.579724 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.579727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.579832 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.579950 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.580077 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":86,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:41:49.592135 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.583540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.98.61:35707)
0927 00:41:49.583647 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.583651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.583966 (+ 315us) server_negotiation.cc:407] Connection header received
0927 00:41:49.584134 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.584137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.584184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.584280 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.584817 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.585558 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.586413 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.586733 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.586838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.589047 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.589051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.589054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.589078 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.590510 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.590925 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.590928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.590930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.591009 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.591250 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.591253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.591255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.591443 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.591533 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.591950 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.592034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:49.592227 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.583506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48364)
0927 00:41:49.583845 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.583889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.584012 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.584306 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.584310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.584622 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.584713 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.584718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.585657 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.585659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.586312 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.586321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.586838 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.586840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.586913 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.587404 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.587415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.588861 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.590618 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.590625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.590629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.590834 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.591097 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.591099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.591100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.591171 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.591588 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.591618 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.591738 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.591887 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.592065 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":118,"threads_started":1}
I20260927 00:41:49.593747 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35707
}
attrs {
promote: true
}
}
dest_uuid: "f680812cde3b440791bed40f5e7a9744"
cas_config_opid_index: -1
I20260927 00:41:49.594321 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } }
I20260927 00:41:49.596078 7072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } }
I20260927 00:41:49.596640 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:49.597783 7072 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35707
I20260927 00:41:49.597993 7018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 00:41:49.597999 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } }
I20260927 00:41:49.598121 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:49.598675 6976 master.cc:537] Master@127.0.98.61:35707 shutting down...
I20260927 00:41:49.598718 7072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:41:49.602421 6499 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 00:41:49.603341 6976 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:49.603557 6976 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:41:49.603535 6499 consensus_peers.cc:487] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 -> Peer 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61:35707): Couldn't send request to peer 60a5cbbba72f4a0eba57906c09cf85a4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:41:49.603633 6976 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 00:41:49.618206 6976 master.cc:552] Master@127.0.98.61:35707 shutdown complete.
I20260927 00:41:49.618389 6976 master_runner.cc:305] Clearing existing system tablet
I20260927 00:41:49.618909 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.619057 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.621037 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:49.621919 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.622148 6976 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.622269 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:49.622300 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "f249e8134d42ce7f136c068ce1dc9416"
server_key_iv: "99ddc80c1c8a18b93be8619639e1de4d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.624735 6976 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 00:41:49.626000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37343
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:49.627041 6976 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:41:49.627159 6976 log.cc:1196] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 00:41:49.627430 6976 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Deleting consensus metadata
I20260927 00:41:49.627607 6976 master_runner.cc:315] Copying system tablet from 127.0.98.62:37431
I20260927 00:41:49.628829 6976 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37431
I20260927 00:41:49.638561 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.629468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47336)
0927 00:41:49.629743 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.629757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.629886 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.630176 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.630179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.630400 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.630520 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.630527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.631725 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.631728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.632990 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.633001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.633697 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.633700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.633793 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.634391 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.634407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.634803 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.636925 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.636928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.636931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.637167 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.637483 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.637486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.637488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.637578 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.638042 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.638045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.638136 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.638239 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.638373 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260927 00:41:49.638653 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.629544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.0.98.62:37431)
0927 00:41:49.629697 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.629704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.629860 (+ 156us) server_negotiation.cc:407] Connection header received
0927 00:41:49.629973 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.629979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.630043 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.630151 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.630656 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.631594 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.633134 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.633560 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.633706 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.634945 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.634950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.634956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.634988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.636789 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.637272 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.637278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.637283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.637381 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.637678 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.637684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.637688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.637936 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.638046 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.638309 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.638415 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:41:49.639057 6532 tablet_copy_service.cc:140] P f680812cde3b440791bed40f5e7a9744: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60a5cbbba72f4a0eba57906c09cf85a4 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47336)
I20260927 00:41:49.639220 6532 tablet_copy_service.cc:161] P f680812cde3b440791bed40f5e7a9744: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60a5cbbba72f4a0eba57906c09cf85a4 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47336: session id = 60a5cbbba72f4a0eba57906c09cf85a4-00000000000000000000000000000000
I20260927 00:41:49.640849 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:41:49.642472 6976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:41:49.644624 6976 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: tablet copy: Starting download of 0 data blocks...
I20260927 00:41:49.644807 6976 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: tablet copy: Starting download of 1 WAL segments...
I20260927 00:41:49.646519 6976 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:41:49.648190 6532 tablet_copy_service.cc:342] P f680812cde3b440791bed40f5e7a9744: Request end of tablet copy session 60a5cbbba72f4a0eba57906c09cf85a4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47336
I20260927 00:41:49.648303 6532 tablet_copy_service.cc:433] P f680812cde3b440791bed40f5e7a9744: ending tablet copy session 60a5cbbba72f4a0eba57906c09cf85a4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60a5cbbba72f4a0eba57906c09cf85a4
I20260927 00:41:49.656804 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:49.658478 7085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.661419 7086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.661856 6976 server_base.cc:1034] running on GCE node
W20260927 00:41:49.665036 7088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.665295 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.665594 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.666738 6976 hybrid_clock.cc:648] HybridClock initialized: now 1790469709666715 us; error 41 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.668372 6976 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 00:41:49.668857 6976 webserver.cc:460] Webserver started at http://127.0.98.61:42855/ using document root <none> and password file <none>
I20260927 00:41:49.669081 6976 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.669124 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.670393 6976 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:49.671383 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.671663 6976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:49.671748 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "f249e8134d42ce7f136c068ce1dc9416"
server_key_iv: "99ddc80c1c8a18b93be8619639e1de4d"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.671828 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.692397 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:49.694710 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:49.699512 6976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35707
I20260927 00:41:49.699620 7147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35707 every 8 connection(s)
I20260927 00:41:49.700062 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:41:49.701903 7148 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:41:49.702957 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Bootstrap starting.
I20260927 00:41:49.713476 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:49.713788 7148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: Bootstrap complete.
I20260927 00:41:49.714357 7148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } }
I20260927 00:41:49.714495 7148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60a5cbbba72f4a0eba57906c09cf85a4, State: Initialized, Role: LEARNER
I20260927 00:41:49.714596 7148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } }
I20260927 00:41:49.714910 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } }
I20260927 00:41:49.715018 7148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:49.715094 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:41:49.718051 7162 catalog_manager.cc:1259] Loaded cluster ID: 8a74c60375024ff18669926adc5eea10
I20260927 00:41:49.718106 7162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: loading cluster ID for follower catalog manager: success
I20260927 00:41:49.720099 7162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: acquiring CA information for follower catalog manager: success
I20260927 00:41:49.721397 7162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:41:49.757961 7078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:49.758188 7078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:49.758224 7078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:49.761958 7078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:49.761999 7078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:49.762018 7078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:49.762034 7078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:49.762049 7078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:49.767788 7078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:49.768640 7078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:49.769595 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:49.777838 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.777899 7170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.777864 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.778092 7078 server_base.cc:1034] running on GCE node
I20260927 00:41:49.778621 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.779635 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.780823 7078 hybrid_clock.cc:648] HybridClock initialized: now 1790469709780803 us; error 36 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.783360 7078 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:41:49.784149 7078 webserver.cc:460] Webserver started at http://127.0.98.60:34317/ using document root <none> and password file <none>
I20260927 00:41:49.784660 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.784751 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.785025 7078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:49.786974 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/instance:
uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "54fbaf6d99bfb2118ce879ee2330591e"
server_key_iv: "9ab7fa15e84483c36d34ad931e7bf1e3"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.787506 7078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal/instance:
uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "54fbaf6d99bfb2118ce879ee2330591e"
server_key_iv: "9ab7fa15e84483c36d34ad931e7bf1e3"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.790998 7078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:41:49.793283 7177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.794467 7078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:49.794584 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "54fbaf6d99bfb2118ce879ee2330591e"
server_key_iv: "9ab7fa15e84483c36d34ad931e7bf1e3"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.794683 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.806211 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:49.808692 7078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:49.808872 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:49.815547 7078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37343
I20260927 00:41:49.815806 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37343 every 8 connection(s)
I20260927 00:41:49.816633 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:41:49.819293 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:49.822926 7230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:41:49.824096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7078
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:41:49.839123 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.825320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.62:37431)
0927 00:41:49.825462 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.825466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.826141 (+ 675us) server_negotiation.cc:407] Connection header received
0927 00:41:49.827422 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.827427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.827509 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.827603 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.828826 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.829719 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.831119 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.831461 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.831560 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.835261 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.835265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.835267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.835297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.836978 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.837600 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.837603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.837604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.837672 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.838000 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.838005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.838006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.838177 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.838289 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.838793 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.838945 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:41:49.840135 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.825373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47346)
0927 00:41:49.825962 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.826033 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.827170 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.827805 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.827813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.828293 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.828671 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.828700 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.829865 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.829868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.830995 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.831005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.831589 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.831593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.831824 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.832673 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.832696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.835094 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.837086 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.837096 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.837106 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.837512 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.837766 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.837771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.837773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.837908 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.838299 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.838306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.838562 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.838740 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.839039 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":131,"threads_started":1}
I20260927 00:41:49.840137 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.826305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.98.60:37343)
0927 00:41:49.826599 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.826610 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.826652 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:41:49.827170 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.827193 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.827446 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.827618 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.828894 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.830391 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.831456 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.832075 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.832187 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.835742 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.835748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.835760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.835787 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.837551 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.837973 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.837978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.837980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.838060 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.838324 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.838329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.838330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.838632 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.839375 (+ 743us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.839465 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.839599 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"spinlock_wait_cycles":768,"thread_start_us":87,"threads_started":1}
I20260927 00:41:49.841542 7230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:41:49.849745 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.842460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48366)
0927 00:41:49.842757 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.842768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.842824 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.843018 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.843020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.843137 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.843231 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.843235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.844100 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.844102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.844618 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.844625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.845119 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.845121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.845199 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.845627 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.845637 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.846766 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.848485 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.848488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.848490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.848678 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.849006 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.849009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.849010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.849084 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.849404 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.849406 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.849471 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.849547 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.849636 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260927 00:41:49.849844 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.842586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.0.98.61:35707)
0927 00:41:49.842791 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.842794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.842806 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:41:49.842888 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.842890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.842926 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.842994 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.843322 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.844007 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.844714 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.845034 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.845118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.846901 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.846904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.846906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.846927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.848353 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.848786 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.848791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.848793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.848872 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.849163 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.849165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.849166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.849325 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.849402 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.849589 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.849679 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260927 00:41:49.850533 7230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:41:49.857475 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.850997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44720)
0927 00:41:49.851092 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.851103 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.851159 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.851376 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.851378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.851492 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.851551 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.851556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.852313 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.852316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.852781 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.852788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.853263 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.853266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.853320 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.853744 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.853755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.854872 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.856269 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.856272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.856273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.856443 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.856661 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.856663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.856664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.856740 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.857139 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.857141 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.857208 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.857287 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.857369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 00:41:49.857604 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.851119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.0.98.60:37343)
0927 00:41:49.851203 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.851207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.851221 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:49.851253 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.851256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.851292 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.851354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.851645 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.852225 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.852874 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.853172 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.853263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.854997 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.854999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.855001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.855021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.856179 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.856522 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.856524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.856525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.856589 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.856814 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.856816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.856817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.857043 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.857153 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.857352 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.857434 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 00:41:49.860494 7230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Bootstrap starting.
I20260927 00:41:49.862839 7230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:49.863554 7230 log.cc:824] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:49.865437 7230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: No bootstrap required, opened a new log
I20260927 00:41:49.868220 7230 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } }
I20260927 00:41:49.868458 7230 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:49.868535 7230 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bc509975fc4a70a2a4cf29f839bb33, State: Initialized, Role: FOLLOWER
I20260927 00:41:49.869022 7230 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } }
I20260927 00:41:49.869468 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } } }
I20260927 00:41:49.869589 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:49.870196 7230 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 00:41:49.875032 7247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:49.875113 7247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:49.878042 7078 master_runner.cc:428] Detected that this master b4bc509975fc4a70a2a4cf29f839bb33 is joining an existing cluster
I20260927 00:41:49.878108 7078 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37343
I20260927 00:41:49.889441 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.0.98.62:37431)
0927 00:41:49.880612 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.880616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.880858 (+ 242us) server_negotiation.cc:407] Connection header received
0927 00:41:49.881078 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.881083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.881134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.881185 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.882130 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.882938 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884413 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884755 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884825 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.886362 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.886366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.886368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.886393 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.887845 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.888245 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.888251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.888253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.888330 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.888610 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.888615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.888616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.888787 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.888884 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.889197 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.889304 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:49.889460 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47354)
0927 00:41:49.880779 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.880793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.880963 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.881325 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.881328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.881695 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.881999 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.882005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.883076 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.883079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.884282 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.884291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884994 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.885418 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.885850 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.885862 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.886219 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.887942 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.887948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.887951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.888144 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.888436 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.888439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.888441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.888523 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.888884 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.888913 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.889024 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.889134 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.889266 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":613,"spinlock_wait_cycles":1280,"thread_start_us":136,"threads_started":1}
I20260927 00:41:49.893879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6574
I20260927 00:41:49.894124 6703 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469708, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 00:41:49.900607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34025
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37431
--webserver_port=39463
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:49.917668 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.893111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44744)
0927 00:41:49.893593 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.893608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.893699 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.894035 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.894039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.894221 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.894318 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.894325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.895447 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.895450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.896305 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.896316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.896968 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.896975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.897124 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.897617 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.897637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.899248 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.915887 (+ 16639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.915891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.915894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.916177 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.916577 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.916583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.916671 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.917141 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.917144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.917241 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.917323 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.917490 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384}
I20260927 00:41:49.917829 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48370)
0927 00:41:49.880878 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.880924 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.881042 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.881382 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.881388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.881689 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.881997 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.882005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.883076 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.883079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.884282 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.884291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884994 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.885134 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.913475 (+ 28341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.913493 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.914039 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.916188 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.916193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.916446 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.916751 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.916756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.916831 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.917297 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.917300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.917386 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.917479 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.917628 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":215,"spinlock_wait_cycles":3456,"thread_start_us":85,"threads_started":1}
I20260927 00:41:49.918340 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.893170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.0.98.60:37343)
0927 00:41:49.893441 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.893449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.893658 (+ 209us) server_negotiation.cc:407] Connection header received
0927 00:41:49.893823 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.893827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.893878 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.894424 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.894517 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.895320 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.896445 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.896802 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.896918 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.899409 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.899416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.899422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.899452 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.915736 (+ 16284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916377 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.916381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.916383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.916473 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916785 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.916788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.916790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.917030 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.917991 (+ 961us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.918064 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.918178 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260927 00:41:49.918624 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.61:35707)
0927 00:41:49.880705 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.880709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.880980 (+ 271us) server_negotiation.cc:407] Connection header received
0927 00:41:49.881137 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.881140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.881198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.881241 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.882130 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.882938 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884413 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884754 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884825 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.914253 (+ 29428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.914258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.914261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.914293 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.915979 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916565 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.916569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.916571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.916659 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.916966 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.916969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.916971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.917172 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.917256 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.918321 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.918461 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:41:49.919265 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37343: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37343
}
attrs {
promote: true
}
}
dest_uuid: "f680812cde3b440791bed40f5e7a9744"
cas_config_opid_index: 5
I20260927 00:41:49.919895 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } }
I20260927 00:41:49.921156 7072 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.922581 7204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:49.922763 7072 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37343
I20260927 00:41:49.922914 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.922995 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:49.923597 7078 master.cc:537] Master@127.0.98.60:37343 shutting down...
I20260927 00:41:49.923903 7204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:41:49.926108 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.921421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48382)
0927 00:41:49.921658 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.921672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.921742 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.922113 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.922117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.922133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:49.922216 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.922221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.923192 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.923196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.924432 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.924443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.925709 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.925712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.925802 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.925843 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.925918 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.925980 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 00:41:49.926514 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.921647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.98.61:35707)
0927 00:41:49.921764 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.921768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.921781 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:49.921819 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.921822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.921875 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.921975 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:49.922335 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.923062 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.924584 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.925354 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.925488 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.925648 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.926332 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.926404 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:41:49.926889 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 LEARNER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 00:41:49.927289 7072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:49.927956 7072 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 00:41:49.929867 7078 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 00:41:49.930001 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 -> Peer b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60:37343): Couldn't send request to peer b4bc509975fc4a70a2a4cf29f839bb33. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 00:41:49.930138 7078 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:49.930261 7078 tablet_replica.cc:330] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: stopping tablet replica
I20260927 00:41:49.930853 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.0.98.60:37343)
0927 00:41:49.880549 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.880553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.880934 (+ 381us) server_negotiation.cc:407] Connection header received
0927 00:41:49.881080 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.881083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.881120 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.881165 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.882263 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.883006 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884209 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884521 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884590 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.919865 (+ 35275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.919869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.919873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.919899 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.926685 (+ 6786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.929339 (+ 2654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.929343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.929345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.929436 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.929749 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.929756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.929761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.929989 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.930070 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.930564 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.930672 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:41:49.930959 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.931531 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.880100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44734)
0927 00:41:49.880820 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.880833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.880987 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.881286 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.881289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.881559 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.882138 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.882147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.883130 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.883133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.884083 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.884094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.884735 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.884738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.884843 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.885385 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.885401 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.915099 (+ 29698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.928932 (+ 13833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.928935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.928937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.929216 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.929545 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.929548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.929550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.929641 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.930251 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.930256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.930406 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.931186 (+ 780us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.931383 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":80,"threads_started":1}
I20260927 00:41:49.933475 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f680812cde3b440791bed40f5e7a9744. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.933599 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:41:49.933694 7151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: true } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.933765 7151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: LEARNER
W20260927 00:41:49.933861 7180 connection.cc:441] server connection from 127.0.0.1:44734 torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:44734 (request call id 0) could send its response
I20260927 00:41:49.936293 7072 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: attempting to promote NON_VOTER 60a5cbbba72f4a0eba57906c09cf85a4 to VOTER
I20260927 00:41:49.936405 7078 master.cc:552] Master@127.0.98.60:37343 shutdown complete.
I20260927 00:41:49.936486 7078 master_runner.cc:305] Clearing existing system tablet
I20260927 00:41:49.936753 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.936689 7072 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } }
I20260927 00:41:49.936877 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.937861 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.937589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44756)
0927 00:41:49.937733 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.937795 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 00:41:49.938091 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 LEARNER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 00:41:49.938496 7071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 00:41:49.938673 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.938451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44770)
0927 00:41:49.938573 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.938613 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 00:41:49.938843 6496 consensus_peers.cc:487] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 -> Peer b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60:37343): Couldn't send request to peer b4bc509975fc4a70a2a4cf29f839bb33. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37343: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:41:49.939842 7078 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:41:49.940373 7071 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.940773 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.941038 7078 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:49.941144 7078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:49.941174 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "54fbaf6d99bfb2118ce879ee2330591e"
server_key_iv: "9ab7fa15e84483c36d34ad931e7bf1e3"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.942103 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.942198 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:49.942580 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.942662 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:49.943082 7078 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:49.943197 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.944846 7260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:49.944980 7260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:49.945487 7078 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 00:41:49.945575 7078 log.cc:1196] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 00:41:49.945809 7078 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Deleting consensus metadata
I20260927 00:41:49.945964 7078 master_runner.cc:315] Copying system tablet from 127.0.98.62:37431
I20260927 00:41:49.947029 7078 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37431
I20260927 00:41:49.956516 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.947623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.0.98.62:37431)
0927 00:41:49.947723 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:49.947728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:49.947950 (+ 222us) server_negotiation.cc:407] Connection header received
0927 00:41:49.948049 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:49.948053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:49.948103 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:49.948180 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:49.948651 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.949502 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.950641 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.951009 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.951111 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.953081 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:49.953085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:49.953088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:49.953132 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:49.954803 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.955287 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.955290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.955293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.955376 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:49.955652 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:49.955656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:49.955657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:49.955864 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:49.955961 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:49.956242 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 00:41:49.956361 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:41:49.956543 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:49.947554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47368)
0927 00:41:49.947849 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:49.947863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:49.947972 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:49.948214 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:49.948217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:49.948440 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:49.948537 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.948544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.949629 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.949633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.950502 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:49.950512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:49.951785 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:49.951788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:49.951878 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:49.952419 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:49.952434 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:49.952920 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:49.954914 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.954917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.954919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.955172 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.955468 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:49.955471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:49.955473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:49.955561 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:49.955965 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:49.955968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:49.956068 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 00:41:49.956178 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 00:41:49.956363 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
I20260927 00:41:49.957192 6532 tablet_copy_service.cc:140] P f680812cde3b440791bed40f5e7a9744: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b4bc509975fc4a70a2a4cf29f839bb33 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47368)
I20260927 00:41:49.957286 6532 tablet_copy_service.cc:161] P f680812cde3b440791bed40f5e7a9744: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b4bc509975fc4a70a2a4cf29f839bb33 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47368: session id = b4bc509975fc4a70a2a4cf29f839bb33-00000000000000000000000000000000
I20260927 00:41:49.958626 6532 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 00:41:49.959841 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 00:41:49.962013 7078 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: tablet copy: Starting download of 0 data blocks...
I20260927 00:41:49.962191 7078 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: tablet copy: Starting download of 1 WAL segments...
I20260927 00:41:49.963534 7078 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 00:41:49.965088 6532 tablet_copy_service.cc:342] P f680812cde3b440791bed40f5e7a9744: Request end of tablet copy session b4bc509975fc4a70a2a4cf29f839bb33-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47368
I20260927 00:41:49.965160 6532 tablet_copy_service.cc:433] P f680812cde3b440791bed40f5e7a9744: ending tablet copy session b4bc509975fc4a70a2a4cf29f839bb33-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b4bc509975fc4a70a2a4cf29f839bb33
I20260927 00:41:49.972975 7078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:49.975164 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.977754 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:49.977865 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:49.978075 7078 server_base.cc:1034] running on GCE node
I20260927 00:41:49.978291 7078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:49.978591 7078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:49.979733 7078 hybrid_clock.cc:648] HybridClock initialized: now 1790469709979728 us; error 38 us; skew 500 ppm
Sep 27 00:41:49 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469709, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:49.981289 7078 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 00:41:49.981742 7078 webserver.cc:460] Webserver started at http://127.0.98.60:38813/ using document root <none> and password file <none>
I20260927 00:41:49.981927 7078 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:49.981968 7078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:49.983209 7078 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:49.984184 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:49.984453 7078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:49.984556 7078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "54fbaf6d99bfb2118ce879ee2330591e"
server_key_iv: "9ab7fa15e84483c36d34ad931e7bf1e3"
server_key_version: "encryptionkey@0"
I20260927 00:41:49.984673 7078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:50.000305 7078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:50.002724 7078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:50.008296 7078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37343
I20260927 00:41:50.008361 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37343 every 8 connection(s)
I20260927 00:41:50.009044 7078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:41:50.011023 7334 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:41:50.012029 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Bootstrap starting.
I20260927 00:41:50.023319 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:41:50.023612 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Bootstrap complete.
I20260927 00:41:50.024178 7334 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } }
I20260927 00:41:50.024338 7334 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4bc509975fc4a70a2a4cf29f839bb33, State: Initialized, Role: LEARNER
I20260927 00:41:50.024458 7334 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } }
I20260927 00:41:50.024780 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:50.024935 7334 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:50.025015 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:41:50.029080 7348 catalog_manager.cc:1259] Loaded cluster ID: 8a74c60375024ff18669926adc5eea10
I20260927 00:41:50.029137 7348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: loading cluster ID for follower catalog manager: success
I20260927 00:41:50.031177 7348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: acquiring CA information for follower catalog manager: success
I20260927 00:41:50.032505 7348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 00:41:50.041869 7257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:50.042122 7257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:50.042193 7257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:50.046324 7257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:50.046378 7257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:50.046463 7257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:50.051959 7257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39463
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:50.053166 7257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:50.054172 7257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:50.063530 7356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.063544 7354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.063602 7353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:50.063740 7257 server_base.cc:1034] running on GCE node
I20260927 00:41:50.064152 7257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:50.064769 7257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:50.065946 7257 hybrid_clock.cc:648] HybridClock initialized: now 1790469710065933 us; error 34 us; skew 500 ppm
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:50.068616 7257 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:41:50.069559 7257 webserver.cc:460] Webserver started at http://127.0.98.1:39463/ using document root <none> and password file <none>
I20260927 00:41:50.070101 7257 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:50.070169 7257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:50.074059 7257 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:50.075911 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.077172 7257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:50.077294 7257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "0b58ea492466483d909e1674d38a9a2d"
format_stamp: "Formatted at 2026-09-27 00:41:48 on dist-test-slave-pkws"
server_key: "5ac121a3a6b178515282a9238dd0d841"
server_key_iv: "bcd1fd7bcc867b47f4a3a30f17dbdbbc"
server_key_version: "encryptionkey@0"
I20260927 00:41:50.077808 7257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:50.091910 6837 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 00:41:50.092729 7257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:50.095405 7257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:50.095541 7257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:50.096027 7257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:50.096797 7257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:50.096858 7257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.096942 7257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:50.096977 7257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.105839 7257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34025
I20260927 00:41:50.106304 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34025 every 8 connection(s)
I20260927 00:41:50.106889 7257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:50.107123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7257
I20260927 00:41:50.107339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6708
I20260927 00:41:50.114892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41039
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37431
--webserver_port=43123
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:50.125921 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.110637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35061 (local address 127.0.98.61:35707)
0927 00:41:50.110773 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.110779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.111158 (+ 379us) server_negotiation.cc:407] Connection header received
0927 00:41:50.111858 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.111864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.111935 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.111992 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.114010 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.114897 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.116995 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117332 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117411 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.121489 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.121493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.121496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.121532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.123352 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.124020 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.124026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.124033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.124122 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.124570 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.124574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.124578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.124812 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.124925 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.125596 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.125720 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:41:50.126845 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.110530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38645 (local address 127.0.98.60:37343)
0927 00:41:50.110775 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.110780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.111392 (+ 612us) server_negotiation.cc:407] Connection header received
0927 00:41:50.111939 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.111943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.111999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.112075 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.114016 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.115113 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117026 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117432 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117546 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.122205 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.122210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.122213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.122259 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.124162 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.124713 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.124719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.124721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.124859 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125305 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.125311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.125314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.125559 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.125660 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.126470 (+ 810us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.126615 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260927 00:41:50.127420 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.108543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53505 (local address 127.0.98.62:37431)
0927 00:41:50.108691 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.108698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.110844 (+ 2146us) server_negotiation.cc:407] Connection header received
0927 00:41:50.111871 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.111875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.111934 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.111988 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.114015 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.114883 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117013 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117340 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117426 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.122934 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.122938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.122941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.122970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.124995 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125447 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.125451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.125452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.125541 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125850 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.125853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.125855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.126061 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.126154 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.127144 (+ 990us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.127251 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:41:50.127997 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.109035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:53505)
0927 00:41:50.110372 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.110430 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.111614 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.112315 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.112324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.112999 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.113850 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.113868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.115271 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.115274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.116838 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.116849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117570 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.117941 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.118862 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.118878 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.122330 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.125095 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.125100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.125335 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.125639 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.125644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.125726 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.126307 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.126310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.126403 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.126493 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.126590 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1119,"mutex_wait_us":618,"spinlock_wait_cycles":84864,"thread_start_us":694,"threads_started":1}
I20260927 00:41:50.128127 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.109141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.98.1:35061)
0927 00:41:50.111038 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.111053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.111617 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.112317 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.112327 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.113185 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.113851 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.113870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.115093 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.115098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.116843 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.116849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117570 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.117897 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.118695 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.118720 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.121331 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.123489 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.123497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.123518 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.123897 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.124249 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.124254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.124257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.124454 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.125013 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.125022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.125323 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.126125 (+ 802us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.126534 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"mutex_wait_us":380,"spinlock_wait_cycles":29696,"thread_start_us":148,"threads_started":1}
I20260927 00:41:50.127997 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.110189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.98.1:38645)
0927 00:41:50.111295 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.111310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.111615 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.112315 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.112324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.112999 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.113853 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.113868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.115223 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.115229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.116859 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.116872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.117697 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.117702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.117947 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.118740 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.118756 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.122030 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.124312 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.124318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.124321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.124595 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.125025 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.125031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.125035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.125134 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.125711 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.125717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.125863 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.126124 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.127252 (+ 1128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":1177,"spinlock_wait_cycles":6784,"thread_start_us":99,"threads_started":1}
I20260927 00:41:50.129228 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35707
I20260927 00:41:50.129267 7478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:50.129354 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37343
I20260927 00:41:50.129463 7477 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.129463 7479 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.129463 7478 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.130008 7478 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:50.130993 7479 heartbeater.cc:511] Master 127.0.98.61:35707 requested a full tablet report, sending...
I20260927 00:41:50.131017 7477 heartbeater.cc:511] Master 127.0.98.60:37343 requested a full tablet report, sending...
I20260927 00:41:50.131325 6512 ts_manager.cc:194] Re-registered known tserver with Master: 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025)
I20260927 00:41:50.131999 7108 ts_manager.cc:194] Registered new tserver with Master: 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025)
I20260927 00:41:50.132037 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53505
I20260927 00:41:50.132972 7298 ts_manager.cc:194] Registered new tserver with Master: 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025)
W20260927 00:41:50.229169 7485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:50.229424 7485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:50.229491 7485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:50.233534 7485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:50.233603 7485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:50.233687 7485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:50.238731 7485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43123
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:50.239712 7485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:50.240672 7485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:50.249753 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.249769 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.249784 7493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:50.249954 7485 server_base.cc:1034] running on GCE node
I20260927 00:41:50.250453 7485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:50.251039 7485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:50.252210 7485 hybrid_clock.cc:648] HybridClock initialized: now 1790469710252176 us; error 47 us; skew 500 ppm
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:50.254823 7485 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:41:50.255612 7485 webserver.cc:460] Webserver started at http://127.0.98.2:43123/ using document root <none> and password file <none>
I20260927 00:41:50.256167 7485 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:50.256237 7485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:50.259701 7485 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:50.261627 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.262764 7485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:50.262874 7485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "40e44f10c2ec43a792344daf15ca4d52"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "2fd2775f585914ae8e2a7b855b1f05c7"
server_key_iv: "ea2e9f947fc91ea6f72edbdd4f4f3770"
server_key_version: "encryptionkey@0"
I20260927 00:41:50.263360 7485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:50.281008 7485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:50.284056 7485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:50.284250 7485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:50.284727 7485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:50.285600 7485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:50.285670 7485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.285734 7485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:50.285782 7485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.295075 7485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41039
I20260927 00:41:50.295150 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41039 every 8 connection(s)
I20260927 00:41:50.296023 7485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:50.299515 6971 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 00:41:50.301793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7485
I20260927 00:41:50.302007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6842
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 00:41:50.311030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43025
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37431
--webserver_port=44489
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:41:50.313558 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.298051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44647 (local address 127.0.98.60:37343)
0927 00:41:50.298167 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.298172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.299230 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 00:41:50.300105 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.300109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.300169 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.300257 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.301970 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.302823 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.304925 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305372 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305468 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.308975 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.308982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.308989 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.309026 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.310783 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.311771 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.311775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.311777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.311864 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312252 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.312255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.312257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.312499 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.312569 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.313270 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.313397 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:41:50.313558 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.297743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43509 (local address 127.0.98.62:37431)
0927 00:41:50.297872 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.297876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.298807 (+ 931us) server_negotiation.cc:407] Connection header received
0927 00:41:50.299950 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.299957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.300012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.300088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.301888 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.302694 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.304865 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305222 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.309036 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.309041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.309044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.309078 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.311803 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312265 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.312268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.312270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.312345 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312650 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.312653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.312655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.312844 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.312936 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.313281 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.313401 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:41:50.314697 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.297812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46361 (local address 127.0.98.61:35707)
0927 00:41:50.298064 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.298069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.299300 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 00:41:50.299991 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.299995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.300043 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.300097 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.301889 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.302694 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.304865 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305222 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.310423 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.310428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.310432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.310462 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.312168 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312708 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.312713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.312716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.312822 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.313170 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.313175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.313177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.313413 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.313490 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.314305 (+ 815us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.314444 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260927 00:41:50.315197 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.298325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.98.2:44647)
0927 00:41:50.299139 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.299150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.299721 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.300404 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.300410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.301048 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.301735 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.301747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.303025 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.303031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.304741 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.304755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305638 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.305854 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.306579 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.306604 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.308757 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.311231 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.311237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.311252 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.311635 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.311972 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.311978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.311981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.312141 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.312648 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.312654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.312999 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.313711 (+ 712us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.314155 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":440,"spinlock_wait_cycles":896,"thread_start_us":158,"threads_started":1}
I20260927 00:41:50.315197 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.298176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:43509)
0927 00:41:50.298644 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.298696 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.299721 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.300639 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.300644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.301048 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.301726 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.301744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.302871 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.302874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.304738 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.304744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305484 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.305791 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.306758 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.306775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.308836 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.311927 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.311933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.311938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.312151 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.312462 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.312467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.312551 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.313005 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.313007 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.313076 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.313796 (+ 720us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.314202 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":680,"spinlock_wait_cycles":4352,"thread_start_us":99,"threads_started":1}
I20260927 00:41:50.316460 7615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:50.316493 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37343
I20260927 00:41:50.316687 7615 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.316689 7614 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.317036 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.298899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.98.2:46361)
0927 00:41:50.299219 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.299233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.299852 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.300405 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.300412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.301048 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.301741 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.301747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.302871 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.302874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.304730 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.304740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.305484 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.305487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.305792 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.306622 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.306638 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.310242 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.312301 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.312312 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.312578 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.312964 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.312970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.312972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.313064 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.313568 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.313571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.314106 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.315104 (+ 998us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.315277 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":587,"spinlock_wait_cycles":17408,"thread_start_us":94,"threads_started":1}
I20260927 00:41:50.317315 7615 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:50.317617 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35707
I20260927 00:41:50.317699 7616 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.317315 7614 heartbeater.cc:511] Master 127.0.98.60:37343 requested a full tablet report, sending...
I20260927 00:41:50.317852 7616 heartbeater.cc:511] Master 127.0.98.61:35707 requested a full tablet report, sending...
I20260927 00:41:50.318391 6512 ts_manager.cc:194] Re-registered known tserver with Master: 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
I20260927 00:41:50.318506 7298 ts_manager.cc:194] Registered new tserver with Master: 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
I20260927 00:41:50.319275 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43509
I20260927 00:41:50.319881 7108 ts_manager.cc:194] Registered new tserver with Master: 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
W20260927 00:41:50.423311 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:50.423561 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:50.423602 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:50.427798 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:50.427850 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:50.427906 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:50.432922 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44489
--tserver_master_addrs=127.0.98.62:37431,127.0.98.61:35707,127.0.98.60:37343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:50.433951 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:50.435024 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:41:50.444841 7621 server_base.cc:1034] running on GCE node
W20260927 00:41:50.444708 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.444698 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:50.444708 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:50.445698 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:50.446277 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:50.447441 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790469710447433 us; error 27 us; skew 500 ppm
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:50.450330 7621 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:41:50.451375 7621 webserver.cc:460] Webserver started at http://127.0.98.3:44489/ using document root <none> and password file <none>
I20260927 00:41:50.452131 7621 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:50.452227 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:50.456203 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:50.458302 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.459441 7621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:50.459570 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
format_stamp: "Formatted at 2026-09-27 00:41:49 on dist-test-slave-pkws"
server_key: "fb645d1788f346c85835cf4da24014f1"
server_key_iv: "5b20cd01d358b1a05c042cea1b3c4b5d"
server_key_version: "encryptionkey@0"
I20260927 00:41:50.460089 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:50.476927 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:50.479869 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.475925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44778)
0927 00:41:50.476154 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.476166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.476254 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.476540 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.476542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.476556 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.476646 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.476650 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.477657 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.477663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.478588 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.478604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.479543 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.479546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.479618 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.479640 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.479692 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.479750 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260927 00:41:50.479913 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.476041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.98.60:37343)
0927 00:41:50.476196 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.476201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.476217 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:50.476358 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.476361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.476423 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.476525 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.476756 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.477523 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.478726 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.479422 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.479533 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.479674 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.479714 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.479789 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:41:50.480238 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:50.480427 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:50.480629 7308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 00:41:50.481040 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f680812cde3b440791bed40f5e7a9744. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:50.481148 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:41:50.481184 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: LEARNER
I20260927 00:41:50.482087 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:50.482134 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.482197 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:50.482237 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:50.491110 7621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43025
I20260927 00:41:50.491155 7751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43025 every 8 connection(s)
I20260927 00:41:50.492012 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:50.498122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7621
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 00:41:50 dist-test-slave-pkws krb5kdc[6467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469710, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:50.510159 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.493989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38521 (local address 127.0.98.61:35707)
0927 00:41:50.494188 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.494194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.495420 (+ 1226us) server_negotiation.cc:407] Connection header received
0927 00:41:50.496196 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.496198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.496240 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.496326 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.497572 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498224 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.499859 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.500086 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.506327 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.506331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.506333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.506362 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.508210 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.508713 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.508718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.508721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.508823 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.509196 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.509199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.509201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.509413 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.509489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.509870 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.509980 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 00:41:50.510183 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.494158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.98.3:60849)
0927 00:41:50.494701 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.494758 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.495758 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.496284 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.496292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.496825 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.497441 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.497450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498530 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.498533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.499646 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.499655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.500301 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.500425 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.501220 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.501239 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.503780 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.506092 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.506099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.506115 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.506988 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.507304 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.507307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.507309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.507471 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.508342 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.508348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.508616 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.508811 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.509156 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":142,"spinlock_wait_cycles":29440,"thread_start_us":146,"threads_started":1}
I20260927 00:41:50.510892 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.494158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.98.3:38521)
0927 00:41:50.495318 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.495332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.495765 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.496343 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.496346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.496824 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.497431 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.497445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498387 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.498390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.499680 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.499690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.500230 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.500413 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.501065 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.501086 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.505238 (+ 4152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.508332 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.508337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.508339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.508588 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.508974 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.508980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.508985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.509079 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.509567 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.509570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.509660 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.510298 (+ 638us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.510450 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":733,"spinlock_wait_cycles":19968,"thread_start_us":93,"threads_started":1}
I20260927 00:41:50.511761 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.494158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:34541)
0927 00:41:50.494701 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.494758 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.495758 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.496284 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.496292 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.496827 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:50.497442 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.497450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498530 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.498533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.500369 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.500378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.501105 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.501109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.501219 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.501669 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:50.501687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:50.504146 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:50.506109 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.506112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.506114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.507699 (+ 1585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.510631 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:50.510636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:50.510638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:50.510740 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:50.511222 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:50.511227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:50.511311 (+ 84us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.511401 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.511535 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":839,"spinlock_wait_cycles":38144,"thread_start_us":117,"threads_started":1}
I20260927 00:41:50.511925 7754 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35707
I20260927 00:41:50.512148 7754 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.512234 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.493884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60849 (local address 127.0.98.60:37343)
0927 00:41:50.494101 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.494105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.494850 (+ 745us) server_negotiation.cc:407] Connection header received
0927 00:41:50.495959 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.495962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.496015 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.496097 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.497589 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498359 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.499767 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.500098 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500188 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.503947 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.503953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.503959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.503991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.505951 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.507114 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.507118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.507119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.507207 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.507977 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.507980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.507981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.508208 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.508292 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.511966 (+ 3674us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.512083 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260927 00:41:50.512662 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.494264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34541 (local address 127.0.98.62:37431)
0927 00:41:50.494496 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.494501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.494849 (+ 348us) server_negotiation.cc:407] Connection header received
0927 00:41:50.495967 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.495970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.496019 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.496074 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.497597 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.498356 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500482 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.500809 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.500951 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.504294 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.504298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.504300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.504326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.505968 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.507796 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.507800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.507802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.507882 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.510855 (+ 2973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.510858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.510860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.511094 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.511179 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.512420 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.512527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 00:41:50.512676 7754 heartbeater.cc:511] Master 127.0.98.61:35707 requested a full tablet report, sending...
I20260927 00:41:50.512975 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.501299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.0.98.60:37343)
0927 00:41:50.502373 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.502382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.502392 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:41:50.502445 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.502448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.502495 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.502602 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.503475 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.504343 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.505593 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.505965 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.506067 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.507794 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.507798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.507800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.507825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.509622 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.511247 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.511253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.511258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.511342 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.511630 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.511633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.511634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.511958 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.512076 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.512665 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.512802 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":514,"threads_started":1}
I20260927 00:41:50.513300 7753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37431
I20260927 00:41:50.513375 7753 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.513515 7753 heartbeater.cc:511] Master 127.0.98.62:37431 requested a full tablet report, sending...
I20260927 00:41:50.513567 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37343
I20260927 00:41:50.513650 7752 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:50.513801 7752 heartbeater.cc:511] Master 127.0.98.60:37343 requested a full tablet report, sending...
I20260927 00:41:50.514200 6512 ts_manager.cc:194] Re-registered known tserver with Master: 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:50.514371 7298 ts_manager.cc:194] Registered new tserver with Master: 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:50.514418 7108 ts_manager.cc:194] Registered new tserver with Master: 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:50.514901 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34541
I20260927 00:41:50.517904 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.505216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.61:35707)
0927 00:41:50.505803 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.505807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.506126 (+ 319us) server_negotiation.cc:407] Connection header received
0927 00:41:50.506164 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.506166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.506215 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.506295 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.507236 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.508003 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.509086 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.509397 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.509491 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.511376 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.511383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.511387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.511412 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.516178 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.516604 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.516630 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.516633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.516718 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.517040 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.517047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.517049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.517276 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.517430 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.517618 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.517733 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":102,"threads_started":1}
I20260927 00:41:50.520960 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.501707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.0.98.62:37431)
0927 00:41:50.502272 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.502275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.503128 (+ 853us) server_negotiation.cc:407] Connection header received
0927 00:41:50.503172 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.503176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.503224 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.503300 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.504045 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.513745 (+ 9700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.514818 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.515207 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.515321 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.517914 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.517916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.517918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.517938 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.519361 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.519776 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.519780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.519781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.519866 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.520133 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.520138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.520140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.520307 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.520440 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.520657 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.520776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":61,"threads_started":1}
I20260927 00:41:50.523274 6512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:50.526989 6512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:50.529474 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: true } } }
I20260927 00:41:50.529620 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:50.532913 7637 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: attempting to promote NON_VOTER b4bc509975fc4a70a2a4cf29f839bb33 to VOTER
I20260927 00:41:50.533254 7637 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:50.534521 7307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 LEARNER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:41:50.534588 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Refusing update from remote peer f680812cde3b440791bed40f5e7a9744: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 00:41:50.534889 7637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 00:41:50.535006 7770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 00:41:50.536437 7770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.537860 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.537958 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:50.537668 7308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.538074 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.538159 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:50.538455 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.538563 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:50.538854 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.539654 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.539755 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:50.543656 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f680812cde3b440791bed40f5e7a9744" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:50.543787 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:50.548065 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.542238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34025 (local address 127.0.0.1:33264)
0927 00:41:50.542465 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.542481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.542554 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.543862 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.543866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.543884 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.543986 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.543993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.545605 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.545616 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.546759 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.546769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.547683 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.547687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.547769 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.547791 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.547869 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.547946 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260927 00:41:50.548169 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.542741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.98.1:34025)
0927 00:41:50.543329 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.543343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.543370 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:41:50.543448 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.543454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.543693 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.544155 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.544307 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.545492 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.546861 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.547519 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.547644 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.547792 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.547927 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.548019 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":96,"threads_started":1}
I20260927 00:41:50.549301 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.542847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41039 (local address 127.0.0.1:34592)
0927 00:41:50.544013 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.544027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.544103 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.545505 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.545508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.545529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.545627 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.545632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.547070 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.547074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.547908 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.547918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.548976 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.548981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.549075 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.549101 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.549152 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.549202 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":15,"thread_start_us":72,"threads_started":1}
I20260927 00:41:50.549763 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.542528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43025 (local address 127.0.0.1:49672)
0927 00:41:50.542760 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.542775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.542862 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.544787 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.544789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.544805 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.544924 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.544934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.546058 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.546062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.548064 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.548070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.549370 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.549373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.549459 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.549478 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.549580 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.549645 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":22,"thread_start_us":110,"threads_started":1}
I20260927 00:41:50.549800 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.543249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.98.3:43025)
0927 00:41:50.544330 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.544343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.544366 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:41:50.544435 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.544440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.544630 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.544802 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.545065 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.545920 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.548176 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.549061 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.549143 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.549263 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.549619 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.549696 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":106,"threads_started":1}
I20260927 00:41:50.551223 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f (DEFAULT_TABLE table=test-table [id=2424d6aaf0f145028557d93347bc92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:50.552112 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b785bacfd142adaf6e023d0b9d8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:50.553323 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.543199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.98.2:41039)
0927 00:41:50.545095 (+ 1896us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.545105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.545126 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:41:50.545192 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.545197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.545360 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.545748 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.545856 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.546943 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.548045 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.548819 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.548961 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.553091 (+ 4130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.553139 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.553216 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":92,"threads_started":1}
I20260927 00:41:50.555658 7548 tablet_service.cc:1467] Processing CreateTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f (DEFAULT_TABLE table=test-table [id=2424d6aaf0f145028557d93347bc92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:50.555542 7686 tablet_service.cc:1467] Processing CreateTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f (DEFAULT_TABLE table=test-table [id=2424d6aaf0f145028557d93347bc92f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:50.556558 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b785bacfd142adaf6e023d0b9d8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:50.556577 7686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b785bacfd142adaf6e023d0b9d8b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:50.560024 7785 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Bootstrap starting.
I20260927 00:41:50.562481 7785 tablet_bootstrap.cc:654] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:50.563272 7786 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Bootstrap starting.
I20260927 00:41:50.563344 7785 log.cc:824] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:50.564188 7788 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Bootstrap starting.
I20260927 00:41:50.565512 7786 tablet_bootstrap.cc:654] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:50.566239 7786 log.cc:824] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:50.566301 7785 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: No bootstrap required, opened a new log
I20260927 00:41:50.566318 7788 tablet_bootstrap.cc:654] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:50.566494 7785 ts_tablet_manager.cc:1397] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:41:50.567082 7788 log.cc:824] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:50.568102 7786 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: No bootstrap required, opened a new log
I20260927 00:41:50.568313 7786 ts_tablet_manager.cc:1397] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:41:50.569296 7788 tablet_bootstrap.cc:492] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: No bootstrap required, opened a new log
I20260927 00:41:50.569475 7788 ts_tablet_manager.cc:1397] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:41:50.571576 7785 raft_consensus.cc:348] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.571857 7785 raft_consensus.cc:374] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:50.571923 7785 raft_consensus.cc:729] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b58ea492466483d909e1674d38a9a2d, State: Initialized, Role: FOLLOWER
I20260927 00:41:50.572207 7786 raft_consensus.cc:348] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.572440 7786 raft_consensus.cc:374] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:50.572373 7785 consensus_queue.cc:260] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.572501 7786 raft_consensus.cc:729] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e44f10c2ec43a792344daf15ca4d52, State: Initialized, Role: FOLLOWER
I20260927 00:41:50.573000 7786 consensus_queue.cc:260] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.573472 7785 ts_tablet_manager.cc:1428] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 00:41:50.573868 7786 ts_tablet_manager.cc:1428] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:41:50.573850 7788 raft_consensus.cc:348] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.574178 7478 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 00:41:50.574270 7615 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 00:41:50.578831 7788 raft_consensus.cc:374] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:50.578909 7788 raft_consensus.cc:729] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d38aef03f0d4f1280bd40a07d8cb1c2, State: Initialized, Role: FOLLOWER
I20260927 00:41:50.579380 7788 consensus_queue.cc:260] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.580016 7753 heartbeater.cc:503] Master 127.0.98.62:37431 was elected leader, sending a full tablet report...
I20260927 00:41:50.580240 7788 ts_tablet_manager.cc:1428] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
W20260927 00:41:50.608783 7480 tablet.cc:2369] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:50.697788 7792 raft_consensus.cc:482] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:50.698141 7792 raft_consensus.cc:504] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.699318 7792 leader_election.cc:288] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025), 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:50.703748 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.699548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34025 (local address 127.0.98.2:32927)
0927 00:41:50.699731 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.699747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.699850 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.700178 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.700180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.700194 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.700320 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.700326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.701178 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.701184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.702413 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.702423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.703315 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.703318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.703428 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.703455 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.703534 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.703606 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 00:41:50.703907 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.699549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43025 (local address 127.0.98.2:45091)
0927 00:41:50.699993 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:50.700004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:50.700064 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:50.700302 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:50.700306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:50.700325 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:50.700461 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.700468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.701435 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.701438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.702547 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:50.702554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.703454 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.703458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:50.703588 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.703610 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:50.703690 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:41:50.703756 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":290}
I20260927 00:41:50.703783 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.699762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32927 (local address 127.0.98.1:34025)
0927 00:41:50.699873 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.699877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.699894 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:41:50.699961 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.699966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.700022 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.700165 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.700460 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.701065 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.702563 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.703186 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.703311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.703383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.703574 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.703655 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:41:50.704352 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.699682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45091 (local address 127.0.98.3:43025)
0927 00:41:50.699808 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.699813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.700073 (+ 260us) server_negotiation.cc:407] Connection header received
0927 00:41:50.700143 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.700146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.700197 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.700267 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:50.700584 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.701316 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.702666 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.703265 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.703349 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.703398 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.704147 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.704233 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:41:50.704785 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b785bacfd142adaf6e023d0b9d8b2f" candidate_uuid: "40e44f10c2ec43a792344daf15ca4d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b58ea492466483d909e1674d38a9a2d" is_pre_election: true
I20260927 00:41:50.704800 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b785bacfd142adaf6e023d0b9d8b2f" candidate_uuid: "40e44f10c2ec43a792344daf15ca4d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" is_pre_election: true
I20260927 00:41:50.705132 7706 raft_consensus.cc:2454] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e44f10c2ec43a792344daf15ca4d52 in term 0.
I20260927 00:41:50.705133 7431 raft_consensus.cc:2454] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e44f10c2ec43a792344daf15ca4d52 in term 0.
I20260927 00:41:50.705602 7504 leader_election.cc:302] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b58ea492466483d909e1674d38a9a2d, 0d38aef03f0d4f1280bd40a07d8cb1c2, 40e44f10c2ec43a792344daf15ca4d52; no voters:
I20260927 00:41:50.705914 7792 raft_consensus.cc:2781] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:50.705991 7792 raft_consensus.cc:482] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:50.706039 7792 raft_consensus.cc:3037] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:50.707010 7792 raft_consensus.cc:504] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.707388 7792 leader_election.cc:288] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [CANDIDATE]: Term 1 election: Requested vote from peers 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025), 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025)
I20260927 00:41:50.707661 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b785bacfd142adaf6e023d0b9d8b2f" candidate_uuid: "40e44f10c2ec43a792344daf15ca4d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b58ea492466483d909e1674d38a9a2d"
I20260927 00:41:50.707726 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b785bacfd142adaf6e023d0b9d8b2f" candidate_uuid: "40e44f10c2ec43a792344daf15ca4d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
I20260927 00:41:50.707779 7431 raft_consensus.cc:3037] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:50.707880 7706 raft_consensus.cc:3037] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:50.708868 7706 raft_consensus.cc:2454] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e44f10c2ec43a792344daf15ca4d52 in term 1.
I20260927 00:41:50.708964 7431 raft_consensus.cc:2454] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e44f10c2ec43a792344daf15ca4d52 in term 1.
I20260927 00:41:50.709259 7501 leader_election.cc:302] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d38aef03f0d4f1280bd40a07d8cb1c2, 40e44f10c2ec43a792344daf15ca4d52; no voters:
I20260927 00:41:50.709513 7792 raft_consensus.cc:2781] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:50.709733 7792 raft_consensus.cc:686] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 1 LEADER]: Becoming Leader. State: Replica: 40e44f10c2ec43a792344daf15ca4d52, State: Running, Role: LEADER
I20260927 00:41:50.710062 7792 consensus_queue.cc:237] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:50.713991 6511 catalog_manager.cc:5495] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "40e44f10c2ec43a792344daf15ca4d52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:50.733559 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.729825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.0.98.2:41039)
0927 00:41:50.730061 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.730066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.730080 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:50.730185 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.730190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.730261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.730387 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:50.730632 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.731124 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.731964 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.732183 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.732290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.732596 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:50.733049 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:50.733175 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.733348 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.733397 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 00:41:50.739347 7706 raft_consensus.cc:1261] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Refusing update from remote peer 40e44f10c2ec43a792344daf15ca4d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:50.739346 7431 raft_consensus.cc:1261] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Refusing update from remote peer 40e44f10c2ec43a792344daf15ca4d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:50.740032 7792 consensus_queue.cc:1035] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:50.740276 7794 consensus_queue.cc:1035] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 00:41:50.743386 7755 tablet.cc:2369] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:50.748147 7797 mvcc.cc:205] Tried to move back new op lower bound from 7333763935181271040 to 7333763935071154176. Current Snapshot: MvccSnapshot[applied={T|T < 7333763935181271040}]
I20260927 00:41:50.748564 7799 mvcc.cc:205] Tried to move back new op lower bound from 7333763935181271040 to 7333763935071154176. Current Snapshot: MvccSnapshot[applied={T|T < 7333763935181271040}]
I20260927 00:41:50.758623 6511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:50.758790 6511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:50.763212 6511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f680812cde3b440791bed40f5e7a9744: Sending DeleteTablet for 3 replicas of tablet 14b785bacfd142adaf6e023d0b9d8b2f
I20260927 00:41:50.763984 7686 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49672
I20260927 00:41:50.763984 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33264
I20260927 00:41:50.764051 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b785bacfd142adaf6e023d0b9d8b2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:50 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34592
I20260927 00:41:50.764415 7806 tablet_replica.cc:330] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: stopping tablet replica
I20260927 00:41:50.764561 7808 tablet_replica.cc:330] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: stopping tablet replica
I20260927 00:41:50.764730 7806 raft_consensus.cc:2229] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:50.764976 7808 raft_consensus.cc:2229] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:50.765061 7806 raft_consensus.cc:2258] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:50.765447 7808 raft_consensus.cc:2258] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:50.766676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6480
I20260927 00:41:50.768049 7806 ts_tablet_manager.cc:1905] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:50.771467 7806 ts_tablet_manager.cc:1918] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:50.771567 7806 log.cc:1196] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/14b785bacfd142adaf6e023d0b9d8b2f
I20260927 00:41:50.771888 7806 ts_tablet_manager.cc:1939] T 14b785bacfd142adaf6e023d0b9d8b2f P 0b58ea492466483d909e1674d38a9a2d: Deleting consensus metadata
I20260927 00:41:50.773396 7807 tablet_replica.cc:330] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: stopping tablet replica
I20260927 00:41:50.773793 7807 raft_consensus.cc:2229] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:50.774108 7807 raft_consensus.cc:2258] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:41:50.775252 7364 connection.cc:535] server connection from 127.0.0.1:33264 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:41:50.776069 7807 ts_tablet_manager.cc:1905] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:50.778803 7807 ts_tablet_manager.cc:1918] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:50.778898 7807 log.cc:1196] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/14b785bacfd142adaf6e023d0b9d8b2f
I20260927 00:41:50.779213 7807 ts_tablet_manager.cc:1939] T 14b785bacfd142adaf6e023d0b9d8b2f P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting consensus metadata
I20260927 00:41:50.779434 7808 ts_tablet_manager.cc:1905] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:41:50.780265 7639 connection.cc:441] server connection from 127.0.0.1:49672 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49672 (request call id 1) could send its response
I20260927 00:41:50.781689 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.766000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.0.98.61:35707)
0927 00:41:50.766109 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.766125 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.766139 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:50.766174 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.766177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.766222 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.766302 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.772583 (+ 6281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.773425 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.774482 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.774845 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.774937 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.777800 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.777804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.777807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.777834 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.779689 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.780217 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.780221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.780223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.780303 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.780680 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.780683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.780685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.780911 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.781015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.781359 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.781497 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 00:41:50.782332 7808 ts_tablet_manager.cc:1918] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:50.782428 7808 log.cc:1196] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/14b785bacfd142adaf6e023d0b9d8b2f
I20260927 00:41:50.782759 7808 ts_tablet_manager.cc:1939] T 14b785bacfd142adaf6e023d0b9d8b2f P 40e44f10c2ec43a792344daf15ca4d52: Deleting consensus metadata
W20260927 00:41:50.783794 7504 connection.cc:441] server connection from 127.0.0.1:34592 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34592 (request call id 1) could send its response
I20260927 00:41:50.791481 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.781267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.98.61:35707)
0927 00:41:50.781387 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.781391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.782889 (+ 1498us) server_negotiation.cc:407] Connection header received
0927 00:41:50.783084 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.783087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.783132 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.783181 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.784057 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.784695 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.785886 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.786113 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.786221 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.787871 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.787874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.787877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.787899 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.789669 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.790135 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.790142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.790144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.790231 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.790555 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.790559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.790561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.790775 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.790895 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.791233 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.791334 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:50.792572 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.766177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.0.98.60:37343)
0927 00:41:50.766315 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.766319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.783241 (+ 16922us) server_negotiation.cc:407] Connection header received
0927 00:41:50.783397 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.783400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.783450 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.783502 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.784413 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.785228 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.786445 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.786779 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.786899 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.788383 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.788389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.788392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.788422 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.790287 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.790764 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.790767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.790769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.790848 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.791133 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.791137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.791138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.791328 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.791419 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.791973 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.792089 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:41:50.797690 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:50.782245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.0.98.60:37343)
0927 00:41:50.782374 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:50.782378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:50.789499 (+ 7121us) server_negotiation.cc:407] Connection header received
0927 00:41:50.789669 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:50.789672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:50.789721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:50.789774 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:50.790706 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.791423 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.792527 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:50.792793 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:50.792887 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:50.794469 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:50.794472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:50.794475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:50.794498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:50.796150 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.796536 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.796539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.796541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.796631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:50.796863 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:50.796866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:50.796868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:50.797094 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:50.797229 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:50.797437 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 00:41:50.797538 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:51.750967 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.750445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:39329)
0927 00:41:51.750720 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.750812 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
W20260927 00:41:51.751349 7753 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:51.754464 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.754011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:60193)
0927 00:41:51.754255 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.754350 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
W20260927 00:41:51.754755 7478 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:51.755388 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:51.754975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:49683)
0927 00:41:51.755210 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:51.755278 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":51,"threads_started":1}
W20260927 00:41:51.755761 7615 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.263407 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f680812cde3b440791bed40f5e7a9744)
I20260927 00:41:52.263567 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f680812cde3b440791bed40f5e7a9744)
I20260927 00:41:52.263628 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:52.263790 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:52.264678 7821 leader_election.cc:288] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61:35707)
I20260927 00:41:52.264709 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60:37343)
I20260927 00:41:52.265439 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.264824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47928)
0927 00:41:52.265214 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.265295 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":84,"threads_started":1}
W20260927 00:41:52.265630 7098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.265921 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.265026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47932)
0927 00:41:52.265753 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.265818 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":52,"threads_started":1}
W20260927 00:41:52.266096 7284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.266621 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.266331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47946)
0927 00:41:52.266508 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.266556 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260927 00:41:52.266737 7098 leader_election.cc:334] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.267014 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.266727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47962)
0927 00:41:52.266910 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.266965 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260927 00:41:52.267190 7284 leader_election.cc:334] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.269217 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.264871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35707 (local address 127.0.0.1:48434)
0927 00:41:52.265304 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.265319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:52.265426 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:52.265880 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:52.265885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:52.265904 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:52.266015 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:52.266023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.267175 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.267178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:52.268042 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:52.268051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.268745 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:52.268843 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:52.268941 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:41:52.269012 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:41:52.269093 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":7,"thread_start_us":59,"threads_started":1}
I20260927 00:41:52.269294 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.265061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.0.98.61:35707)
0927 00:41:52.265366 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:52.265371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:52.265387 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:52.265620 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:52.265623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:52.265673 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:52.265762 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:52.266134 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.266996 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268164 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.268638 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268717 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:52.268763 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:52.269051 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 00:41:52.269134 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":66,"threads_started":1}
I20260927 00:41:52.269327 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.265358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.60:37343)
0927 00:41:52.265656 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:52.265661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:52.265674 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:52.265717 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:52.265722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:52.265815 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:52.265903 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:52.266362 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.267262 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268355 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.268756 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268869 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:52.268982 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:52.269120 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 00:41:52.269176 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":414,"server-negotiator.queue_time_us":238,"thread_start_us":65,"threads_started":1}
I20260927 00:41:52.269493 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.265035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37343 (local address 127.0.0.1:44806)
0927 00:41:52.265495 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.265509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:52.265583 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:52.265948 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:52.265951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:52.265964 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:52.266146 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:52.266154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.267392 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.267397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:52.268242 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:52.268250 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:52.268857 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:52.268860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:52.268977 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:52.269023 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:41:52.269097 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:41:52.269150 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":56,"threads_started":1}
I20260927 00:41:52.269810 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" is_pre_election: true
I20260927 00:41:52.269887 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" is_pre_election: true
I20260927 00:41:52.270020 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60a5cbbba72f4a0eba57906c09cf85a4 in term 1.
I20260927 00:41:52.270020 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bc509975fc4a70a2a4cf29f839bb33 in term 1.
I20260927 00:41:52.270358 7096 leader_election.cc:302] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60a5cbbba72f4a0eba57906c09cf85a4, b4bc509975fc4a70a2a4cf29f839bb33; no voters: f680812cde3b440791bed40f5e7a9744
I20260927 00:41:52.270363 7285 leader_election.cc:302] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60a5cbbba72f4a0eba57906c09cf85a4, b4bc509975fc4a70a2a4cf29f839bb33; no voters: f680812cde3b440791bed40f5e7a9744
I20260927 00:41:52.270563 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:41:52.270661 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f680812cde3b440791bed40f5e7a9744)
I20260927 00:41:52.270713 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:41:52.270721 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 00:41:52.270807 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f680812cde3b440791bed40f5e7a9744)
I20260927 00:41:52.270855 7821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:41:52.271816 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:52.271904 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:52.272221 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 election: Requested vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), b4bc509975fc4a70a2a4cf29f839bb33 (127.0.98.60:37343)
I20260927 00:41:52.272344 7821 leader_election.cc:288] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 election: Requested vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61:35707)
I20260927 00:41:52.272518 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b4bc509975fc4a70a2a4cf29f839bb33"
I20260927 00:41:52.272590 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.272317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47972)
0927 00:41:52.272473 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.272530 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 00:41:52.272610 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
I20260927 00:41:52.272710 7308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 60a5cbbba72f4a0eba57906c09cf85a4 in current term 2: Already voted for candidate b4bc509975fc4a70a2a4cf29f839bb33 in this term.
I20260927 00:41:52.272742 7114 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b4bc509975fc4a70a2a4cf29f839bb33 in current term 2: Already voted for candidate 60a5cbbba72f4a0eba57906c09cf85a4 in this term.
I20260927 00:41:52.272938 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.272574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47976)
0927 00:41:52.272710 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.272803 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 00:41:52.273581 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.273367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:47984)
0927 00:41:52.273472 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.273522 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 00:41:52.273643 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.273461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:48000)
0927 00:41:52.273538 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.273579 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 00:41:52.273737 7284 leader_election.cc:334] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
W20260927 00:41:52.273737 7098 leader_election.cc:334] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:52.273805 7098 leader_election.cc:302] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60a5cbbba72f4a0eba57906c09cf85a4; no voters: b4bc509975fc4a70a2a4cf29f839bb33, f680812cde3b440791bed40f5e7a9744
I20260927 00:41:52.273805 7284 leader_election.cc:302] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4bc509975fc4a70a2a4cf29f839bb33; no voters: 60a5cbbba72f4a0eba57906c09cf85a4, f680812cde3b440791bed40f5e7a9744
I20260927 00:41:52.273937 7820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:41:52.274044 7821 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 00:41:52.752429 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.751993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:48469)
0927 00:41:52.752278 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.752337 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260927 00:41:52.755733 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.755309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:49405)
0927 00:41:52.755581 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.755644 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260927 00:41:52.756597 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:52.756256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:50411)
0927 00:41:52.756470 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:52.756511 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":57,"threads_started":1}
I20260927 00:41:53.753616 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.753229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:50423)
0927 00:41:53.753477 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.753529 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260927 00:41:53.756961 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.756344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:38547)
0927 00:41:53.756766 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.756841 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":74,"threads_started":1}
I20260927 00:41:53.757658 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:53.757276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:39159)
0927 00:41:53.757505 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:53.757566 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260927 00:41:54.066241 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:54.066362 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:54.066879 7836 leader_election.cc:288] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61:35707)
I20260927 00:41:54.067377 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" is_pre_election: true
I20260927 00:41:54.067482 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.067059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44638)
0927 00:41:54.067331 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.067397 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260927 00:41:54.067543 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4bc509975fc4a70a2a4cf29f839bb33 in term 2.
I20260927 00:41:54.067938 7285 leader_election.cc:302] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60a5cbbba72f4a0eba57906c09cf85a4, b4bc509975fc4a70a2a4cf29f839bb33; no voters:
I20260927 00:41:54.068109 7836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 00:41:54.068189 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:54.068230 7836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:41:54.068373 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.068130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44654)
0927 00:41:54.068250 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.068308 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:41:54.068501 7284 leader_election.cc:334] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:54.069366 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:54.069707 7836 leader_election.cc:288] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 election: Requested vote from peers f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431), 60a5cbbba72f4a0eba57906c09cf85a4 (127.0.98.61:35707)
I20260927 00:41:54.070063 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.069865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44658)
0927 00:41:54.069954 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.070004 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 00:41:54.070046 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60a5cbbba72f4a0eba57906c09cf85a4"
I20260927 00:41:54.070200 7114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:41:54.070668 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.070469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44674)
0927 00:41:54.070570 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.070614 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:41:54.070832 7284 leader_election.cc:334] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
I20260927 00:41:54.071266 7114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4bc509975fc4a70a2a4cf29f839bb33 in term 3.
I20260927 00:41:54.071532 7285 leader_election.cc:302] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60a5cbbba72f4a0eba57906c09cf85a4, b4bc509975fc4a70a2a4cf29f839bb33; no voters: f680812cde3b440791bed40f5e7a9744
I20260927 00:41:54.071713 7836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:41:54.071893 7836 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [term 3 LEADER]: Becoming Leader. State: Replica: b4bc509975fc4a70a2a4cf29f839bb33, State: Running, Role: LEADER
I20260927 00:41:54.072187 7836 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } }
I20260927 00:41:54.073740 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc509975fc4a70a2a4cf29f839bb33. Latest consensus state: current_term: 3 leader_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:54.073855 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:54.074168 7841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:54.076346 7841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:54.076759 7841 catalog_manager.cc:1259] Loaded cluster ID: 8a74c60375024ff18669926adc5eea10
I20260927 00:41:54.076805 7841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:54.077566 7841 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:54.077946 7841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Loaded TSK: 0
I20260927 00:41:54.078612 7841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:41:54.168452 7297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:54.171582 7297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:54.174559 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.174314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44706)
0927 00:41:54.174439 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.174488 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:41:54.174592 7114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [term 3 FOLLOWER]: Refusing update from remote peer b4bc509975fc4a70a2a4cf29f839bb33: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 00:41:54.175081 7836 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 00:41:54.175194 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.175005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.0.1:44720)
0927 00:41:54.175098 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.175141 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:41:54.175324 7284 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 -> Peer f680812cde3b440791bed40f5e7a9744 (127.0.98.62:37431): Couldn't send request to peer f680812cde3b440791bed40f5e7a9744. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 00:41:54.176584 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4bc509975fc4a70a2a4cf29f839bb33. Latest consensus state: current_term: 3 leader_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:54.176685 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:54.177721 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:54.177855 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:54.178058 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f680812cde3b440791bed40f5e7a9744" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37431 } } peers { permanent_uuid: "60a5cbbba72f4a0eba57906c09cf85a4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35707 } attrs { promote: false } } peers { permanent_uuid: "b4bc509975fc4a70a2a4cf29f839bb33" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37343 } attrs { promote: false } } }
I20260927 00:41:54.178175 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a5cbbba72f4a0eba57906c09cf85a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:41:54.187844 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.183489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34025 (local address 127.0.0.1:39216)
0927 00:41:54.183638 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.183651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:54.183735 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:54.184331 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:54.184334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:54.184351 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:54.184486 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.184493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.185601 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.185606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.186612 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.186624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.187459 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.187463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.187553 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.187583 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:41:54.187659 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:54.187727 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 00:41:54.188098 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.183594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.98.1:34025)
0927 00:41:54.184067 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.184073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.184090 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:41:54.184146 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.184150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.184200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.184602 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:54.184718 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.185495 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.186760 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.187336 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.187459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.187523 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.187683 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.187777 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":63,"threads_started":1}
I20260927 00:41:54.188720 7411 tablet_service.cc:1467] Processing CreateTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 (DEFAULT_TABLE table=test-table [id=6daf32f7cf384935aa05fe3629556c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:54.188973 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.184326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.0.98.2:41039)
0927 00:41:54.184556 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.184561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.184750 (+ 189us) server_negotiation.cc:407] Connection header received
0927 00:41:54.184841 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.184846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.184920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.184997 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:54.185364 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.186145 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.187732 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.188375 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.188507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.188567 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.188725 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.188804 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20260927 00:41:54.188983 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.183678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43025 (local address 127.0.0.1:52722)
0927 00:41:54.184227 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.184240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:54.184329 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:54.184919 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:54.184922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:54.184937 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:54.185018 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.185023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.186147 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.186151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.187490 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.187500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.188525 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.188529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.188620 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.188642 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:54.188716 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:41:54.188821 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":160,"thread_start_us":96,"threads_started":1}
I20260927 00:41:54.189049 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc631bf62b0c46aa885ca6fd9d102fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:54.189263 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.184010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41039 (local address 127.0.0.1:40378)
0927 00:41:54.184655 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.184668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:54.184756 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:54.185125 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:54.185128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:54.185145 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:54.185230 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.185237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.186236 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.186239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.187613 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.187620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.188507 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.188511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.188599 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.188620 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:41:54.188709 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:41:54.188804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260927 00:41:54.189458 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.183958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.0.98.3:43025)
0927 00:41:54.184595 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.184600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.184630 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:41:54.184681 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.184683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.184763 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.184881 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:54.185133 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.186032 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.187599 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.188194 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.188324 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.188389 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.189207 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.189278 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":67,"threads_started":1}
I20260927 00:41:54.189985 7686 tablet_service.cc:1467] Processing CreateTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 (DEFAULT_TABLE table=test-table [id=6daf32f7cf384935aa05fe3629556c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:54.190267 7686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc631bf62b0c46aa885ca6fd9d102fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:54.191437 7854 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: Bootstrap starting.
I20260927 00:41:54.191951 7548 tablet_service.cc:1467] Processing CreateTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 (DEFAULT_TABLE table=test-table [id=6daf32f7cf384935aa05fe3629556c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:54.192229 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc631bf62b0c46aa885ca6fd9d102fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:54.192247 7854 tablet_bootstrap.cc:654] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:54.192488 7855 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Bootstrap starting.
I20260927 00:41:54.193409 7855 tablet_bootstrap.cc:654] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:54.193719 7854 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: No bootstrap required, opened a new log
I20260927 00:41:54.193809 7854 ts_tablet_manager.cc:1397] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:54.194272 7854 raft_consensus.cc:348] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.194391 7854 raft_consensus.cc:374] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:54.194442 7854 raft_consensus.cc:729] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b58ea492466483d909e1674d38a9a2d, State: Initialized, Role: FOLLOWER
I20260927 00:41:54.194582 7855 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: No bootstrap required, opened a new log
I20260927 00:41:54.194574 7854 consensus_queue.cc:260] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.194665 7855 ts_tablet_manager.cc:1397] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:54.195073 7855 raft_consensus.cc:348] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.195179 7855 raft_consensus.cc:374] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:54.195277 7854 ts_tablet_manager.cc:1428] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:41:54.195441 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.195215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:39815)
0927 00:41:54.195322 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.195380 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 00:41:54.196074 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.195865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:39325)
0927 00:41:54.195959 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.196010 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 00:41:54.195214 7855 raft_consensus.cc:729] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d38aef03f0d4f1280bd40a07d8cb1c2, State: Initialized, Role: FOLLOWER
I20260927 00:41:54.196249 7855 consensus_queue.cc:260] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.196723 7855 ts_tablet_manager.cc:1428] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:54.197351 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.197124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:38673)
0927 00:41:54.197232 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.197287 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 00:41:54.197377 7477 heartbeater.cc:503] Master 127.0.98.60:37343 was elected leader, sending a full tablet report...
I20260927 00:41:54.197400 7858 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Bootstrap starting.
I20260927 00:41:54.198395 7858 tablet_bootstrap.cc:654] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:54.198467 7752 heartbeater.cc:503] Master 127.0.98.60:37343 was elected leader, sending a full tablet report...
I20260927 00:41:54.199262 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.199049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:35717)
0927 00:41:54.199153 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.199199 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 00:41:54.199570 7858 tablet_bootstrap.cc:492] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: No bootstrap required, opened a new log
I20260927 00:41:54.199673 7858 ts_tablet_manager.cc:1397] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:54.200129 7858 raft_consensus.cc:348] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.200253 7858 raft_consensus.cc:374] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:54.200304 7858 raft_consensus.cc:729] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e44f10c2ec43a792344daf15ca4d52, State: Initialized, Role: FOLLOWER
I20260927 00:41:54.200448 7858 consensus_queue.cc:260] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.200877 7858 ts_tablet_manager.cc:1428] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 00:41:54.201406 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.201169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:55733)
0927 00:41:54.201283 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.201340 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:41:54.202319 7614 heartbeater.cc:503] Master 127.0.98.60:37343 was elected leader, sending a full tablet report...
I20260927 00:41:54.202353 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.202161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:57563)
0927 00:41:54.202243 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.202293 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 00:41:54.281782 7859 raft_consensus.cc:482] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:54.281932 7859 raft_consensus.cc:504] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.283006 7859 leader_election.cc:288] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025), 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
I20260927 00:41:54.286741 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.283220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41039 (local address 127.0.98.1:39281)
0927 00:41:54.283366 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.283383 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:54.283489 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:54.283746 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:54.283749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:54.283785 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:54.283900 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.283905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.284665 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.284670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.285562 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.285571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.286312 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.286316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.286425 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.286443 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 00:41:54.286531 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:41:54.286603 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"spinlock_wait_cycles":12928}
I20260927 00:41:54.286729 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.283358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39281 (local address 127.0.98.2:41039)
0927 00:41:54.283476 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.283480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.283495 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:54.283586 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.283589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.283642 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.283728 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:54.284012 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.284548 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.285700 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.286168 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.286300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.286362 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.286545 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.286611 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:54.286986 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.283335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53689 (local address 127.0.98.3:43025)
0927 00:41:54.283487 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.283493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.283733 (+ 240us) server_negotiation.cc:407] Connection header received
0927 00:41:54.283897 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.283901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.283964 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.284050 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:54.284294 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.284849 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.286022 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.286441 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.286559 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.286615 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.286797 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.286864 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:54.287010 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.283220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43025 (local address 127.0.98.1:53689)
0927 00:41:54.283640 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.283651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:54.283793 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:54.284061 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:54.284064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:54.284078 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:54.284184 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.284189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.284992 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.284998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.285904 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:54.285915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.286564 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.286567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:54.286665 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.286687 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:41:54.286782 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:41:54.286849 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":117,"spinlock_wait_cycles":10240,"thread_start_us":118,"threads_started":1}
I20260927 00:41:54.287297 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc631bf62b0c46aa885ca6fd9d102fa3" candidate_uuid: "0b58ea492466483d909e1674d38a9a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e44f10c2ec43a792344daf15ca4d52" is_pre_election: true
I20260927 00:41:54.287556 7568 raft_consensus.cc:2454] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b58ea492466483d909e1674d38a9a2d in term 0.
I20260927 00:41:54.288004 7365 leader_election.cc:302] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b58ea492466483d909e1674d38a9a2d, 40e44f10c2ec43a792344daf15ca4d52; no voters:
I20260927 00:41:54.288231 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc631bf62b0c46aa885ca6fd9d102fa3" candidate_uuid: "0b58ea492466483d909e1674d38a9a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" is_pre_election: true
I20260927 00:41:54.288290 7859 raft_consensus.cc:2781] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:54.288348 7859 raft_consensus.cc:482] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:54.288388 7859 raft_consensus.cc:3037] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:54.288381 7706 raft_consensus.cc:2454] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b58ea492466483d909e1674d38a9a2d in term 0.
I20260927 00:41:54.289498 7859 raft_consensus.cc:504] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.289880 7859 leader_election.cc:288] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [CANDIDATE]: Term 1 election: Requested vote from peers 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025), 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039)
I20260927 00:41:54.290140 7706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc631bf62b0c46aa885ca6fd9d102fa3" candidate_uuid: "0b58ea492466483d909e1674d38a9a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2"
I20260927 00:41:54.290218 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc631bf62b0c46aa885ca6fd9d102fa3" candidate_uuid: "0b58ea492466483d909e1674d38a9a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e44f10c2ec43a792344daf15ca4d52"
I20260927 00:41:54.290278 7706 raft_consensus.cc:3037] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:54.290357 7568 raft_consensus.cc:3037] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:54.291172 7706 raft_consensus.cc:2454] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b58ea492466483d909e1674d38a9a2d in term 1.
I20260927 00:41:54.291275 7568 raft_consensus.cc:2454] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b58ea492466483d909e1674d38a9a2d in term 1.
I20260927 00:41:54.291446 7364 leader_election.cc:302] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b58ea492466483d909e1674d38a9a2d, 0d38aef03f0d4f1280bd40a07d8cb1c2; no voters:
I20260927 00:41:54.291600 7859 raft_consensus.cc:2781] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:54.291813 7859 raft_consensus.cc:686] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 1 LEADER]: Becoming Leader. State: Replica: 0b58ea492466483d909e1674d38a9a2d, State: Running, Role: LEADER
I20260927 00:41:54.292104 7859 consensus_queue.cc:237] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } }
I20260927 00:41:54.292528 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.292314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:37779)
0927 00:41:54.292402 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.292463 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 00:41:54.294422 7298 catalog_manager.cc:5495] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b58ea492466483d909e1674d38a9a2d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0b58ea492466483d909e1674d38a9a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b58ea492466483d909e1674d38a9a2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 } health_report { overall_health: UNKNOWN } } }
W20260927 00:41:54.299962 7617 tablet.cc:2369] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:54.309798 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.306634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.98.1:34025)
0927 00:41:54.306799 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:54.306809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:54.306822 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:54.306887 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:54.306889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:54.306938 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:54.307028 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:54.307230 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.307713 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.308492 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:54.308711 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:54.308798 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:54.309065 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:54.309496 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:54.309587 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:54.309647 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:41:54.309695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 00:41:54.312515 7706 raft_consensus.cc:1261] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Refusing update from remote peer 0b58ea492466483d909e1674d38a9a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:54.312698 7568 raft_consensus.cc:1261] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Refusing update from remote peer 0b58ea492466483d909e1674d38a9a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:54.313066 7859 consensus_queue.cc:1035] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d38aef03f0d4f1280bd40a07d8cb1c2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:54.313301 7864 consensus_queue.cc:1035] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e44f10c2ec43a792344daf15ca4d52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:54.314779 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.314517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:51529)
0927 00:41:54.314642 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.314707 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 00:41:54.315704 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.315402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:42317)
0927 00:41:54.315552 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.315630 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 00:41:54.319818 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.319336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.1:38905)
0927 00:41:54.319683 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.319743 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279}
I20260927 00:41:54.320492 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.320210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.2:53043)
0927 00:41:54.320362 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.320423 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 00:41:54.320649 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.319987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37431 (local address 127.0.98.3:34187)
0927 00:41:54.320521 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.320581 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463}
I20260927 00:41:54.328990 7297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:54.329164 7297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:54.333209 7297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4bc509975fc4a70a2a4cf29f839bb33: Sending DeleteTablet for 3 replicas of tablet cc631bf62b0c46aa885ca6fd9d102fa3
I20260927 00:41:54.333812 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:54 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39216
I20260927 00:41:54.334136 7686 tablet_service.cc:1514] Processing DeleteTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:54 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52722
I20260927 00:41:54.334167 7877 tablet_replica.cc:330] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d: stopping tablet replica
I20260927 00:41:54.334311 7877 raft_consensus.cc:2229] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:54.334507 7878 tablet_replica.cc:330] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: stopping tablet replica
I20260927 00:41:54.334623 7877 raft_consensus.cc:2258] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0b58ea492466483d909e1674d38a9a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:54.334748 7878 raft_consensus.cc:2229] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:54.334882 7878 raft_consensus.cc:2258] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:54.335268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7257
I20260927 00:41:54.336115 7878 ts_tablet_manager.cc:1905] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:54.337299 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet cc631bf62b0c46aa885ca6fd9d102fa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:54 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40378
I20260927 00:41:54.337644 7879 tablet_replica.cc:330] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: stopping tablet replica
I20260927 00:41:54.337773 7879 raft_consensus.cc:2229] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:54.337913 7879 raft_consensus.cc:2258] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:54.338754 7879 ts_tablet_manager.cc:1905] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:54.339241 7878 ts_tablet_manager.cc:1918] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:54.339326 7878 log.cc:1196] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/cc631bf62b0c46aa885ca6fd9d102fa3
I20260927 00:41:54.339593 7878 ts_tablet_manager.cc:1939] T cc631bf62b0c46aa885ca6fd9d102fa3 P 0d38aef03f0d4f1280bd40a07d8cb1c2: Deleting consensus metadata
I20260927 00:41:54.340379 7282 catalog_manager.cc:4841] TS 0d38aef03f0d4f1280bd40a07d8cb1c2 (127.0.98.3:43025): tablet cc631bf62b0c46aa885ca6fd9d102fa3 (table test-table [id=6daf32f7cf384935aa05fe3629556c7b]) successfully deleted
I20260927 00:41:54.341128 7879 ts_tablet_manager.cc:1918] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:54.341228 7879 log.cc:1196] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/cc631bf62b0c46aa885ca6fd9d102fa3
I20260927 00:41:54.341493 7879 ts_tablet_manager.cc:1939] T cc631bf62b0c46aa885ca6fd9d102fa3 P 40e44f10c2ec43a792344daf15ca4d52: Deleting consensus metadata
I20260927 00:41:54.342078 7283 catalog_manager.cc:4841] TS 40e44f10c2ec43a792344daf15ca4d52 (127.0.98.2:41039): tablet cc631bf62b0c46aa885ca6fd9d102fa3 (table test-table [id=6daf32f7cf384935aa05fe3629556c7b]) successfully deleted
I20260927 00:41:54.343081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7485
I20260927 00:41:54.343205 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:54.342926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34025 (local address 127.0.0.1:39244)
0927 00:41:54.343055 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:54.343115 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:41:54.344148 7285 catalog_manager.cc:4584] TS 0b58ea492466483d909e1674d38a9a2d (127.0.98.1:34025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc631bf62b0c46aa885ca6fd9d102fa3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34025: connect: Connection refused (error 111)
I20260927 00:41:54.349792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7621
I20260927 00:41:54.355095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 6976
I20260927 00:41:54.360685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7078
2026-09-27T00:41:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7110 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:54 dist-test-slave-pkws krb5kdc[7882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:54 dist-test-slave-pkws krb5kdc[7882](info): set up 2 sockets
Sep 27 00:41:54 dist-test-slave-pkws krb5kdc[7882](info): commencing operation
krb5kdc: starting...
W20260927 00:41:55.405630 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:55 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 00:41:55.523882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:55.629454 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:55.629746 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:55.629824 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:55.633797 7895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:55.633881 7895 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 00:41:55.633913 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:55.633975 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:55.634014 7895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:55.634045 7895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:55.639122 7895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40801
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:55.640208 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:55.641417 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:55.649307 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:55.649324 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:55.649324 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:55.649755 7895 server_base.cc:1034] running on GCE node
I20260927 00:41:55.650254 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:55.651197 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:55.652361 7895 hybrid_clock.cc:648] HybridClock initialized: now 1790469715652338 us; error 34 us; skew 500 ppm
Sep 27 00:41:55 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:55.655107 7895 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 00:41:55.655925 7895 webserver.cc:460] Webserver started at http://127.0.98.62:33419/ using document root <none> and password file <none>
I20260927 00:41:55.656474 7895 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:55.656569 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:55.656795 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:55.658804 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "8b692428d1d84c9cb784d19c26a6d8a0"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "a0b4cac8a875419e06c3a4af4c2b3459"
server_key_iv: "cdf6706026b43cb7e948cd08f3a7fd37"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.659354 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b692428d1d84c9cb784d19c26a6d8a0"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "a0b4cac8a875419e06c3a4af4c2b3459"
server_key_iv: "cdf6706026b43cb7e948cd08f3a7fd37"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.662844 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:41:55.665194 7910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:55.666345 7895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:55.666496 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "8b692428d1d84c9cb784d19c26a6d8a0"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "a0b4cac8a875419e06c3a4af4c2b3459"
server_key_iv: "cdf6706026b43cb7e948cd08f3a7fd37"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.666603 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:55.694914 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:55.697942 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:55.698165 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:55.706449 7895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40801
I20260927 00:41:55.706461 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40801 every 8 connection(s)
I20260927 00:41:55.707451 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:55.709142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7895
I20260927 00:41:55.709302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:55.709604 392 external_mini_cluster.cc:1428] Setting key 8a9ee0e2825f6bb42ce98e8566011e73
I20260927 00:41:55.710525 7973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:55.716960 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Bootstrap starting.
Sep 27 00:41:55 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:55.719213 7973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:55.720124 7973 log.cc:824] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:55.722384 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: No bootstrap required, opened a new log
I20260927 00:41:55.724821 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.711017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.0.98.62:40801)
0927 00:41:55.711408 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:55.711426 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:55.711477 (+ 51us) server_negotiation.cc:407] Connection header received
0927 00:41:55.712316 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:55.712337 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:55.712766 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:55.713152 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:55.713838 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.715268 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.716158 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.716815 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.717012 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:55.719517 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:55.719523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:55.719538 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:55.719567 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:55.722148 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.722616 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.722621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.722625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.722731 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.723039 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.723044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.723046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.723428 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:55.723615 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:55.723900 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:41:55.724167 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260927 00:41:55.725329 7973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } }
I20260927 00:41:55.725571 7973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:55.725651 7973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b692428d1d84c9cb784d19c26a6d8a0, State: Initialized, Role: FOLLOWER
I20260927 00:41:55.726114 7973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } }
I20260927 00:41:55.726267 7973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:55.726321 7973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:55.726411 7973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:55.727373 7973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } }
I20260927 00:41:55.727648 7973 leader_election.cc:302] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b692428d1d84c9cb784d19c26a6d8a0; no voters:
I20260927 00:41:55.727870 7973 leader_election.cc:288] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:55.728003 7978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:55.728227 7978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b692428d1d84c9cb784d19c26a6d8a0, State: Running, Role: LEADER
I20260927 00:41:55.728534 7978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } }
I20260927 00:41:55.729115 7973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:55.730214 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b692428d1d84c9cb784d19c26a6d8a0. Latest consensus state: current_term: 1 leader_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } } }
I20260927 00:41:55.730350 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:55.730329 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b692428d1d84c9cb784d19c26a6d8a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40801 } } }
I20260927 00:41:55.730392 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:55.730698 7987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:55.733337 7987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:55.738672 7987 catalog_manager.cc:1347] Generated new cluster ID: bfa44aafc5fe4ac69ba62497e8d7580e
I20260927 00:41:55.738734 7987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:55.748035 7987 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:55.748980 7987 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:55.754163 7987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Generated new TSK 0
I20260927 00:41:55.754823 7987 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 00:41:55.792385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40801
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:55.904258 7999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:55.904568 7999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:55.904650 7999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:55.908609 7999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:55.908663 7999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:55.908742 7999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:55.908807 7999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:41:55.913828 7999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:55.914878 7999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:55.915895 7999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:55.925292 8007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:55.925252 8004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:55.925458 7999 server_base.cc:1034] running on GCE node
W20260927 00:41:55.925369 8005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:55.925989 7999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:55.926608 7999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:55.927798 7999 hybrid_clock.cc:648] HybridClock initialized: now 1790469715927761 us; error 49 us; skew 500 ppm
Sep 27 00:41:55 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:55.930825 7999 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:41:55.931600 7999 webserver.cc:460] Webserver started at http://127.0.98.1:42211/ using document root <none> and password file <none>
I20260927 00:41:55.932106 7999 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:55.932163 7999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:55.932318 7999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:55.934309 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8c7b49d2fca466caff1f946ca06ae07"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "8aebf459c3c2398722995001459f5298"
server_key_iv: "d987fa0b777d5fba3eb02d8611bbf935"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.934803 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8c7b49d2fca466caff1f946ca06ae07"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "8aebf459c3c2398722995001459f5298"
server_key_iv: "d987fa0b777d5fba3eb02d8611bbf935"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.938367 7999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:41:55.940935 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:55.942175 7999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:55.942306 7999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "a8c7b49d2fca466caff1f946ca06ae07"
format_stamp: "Formatted at 2026-09-27 00:41:55 on dist-test-slave-pkws"
server_key: "8aebf459c3c2398722995001459f5298"
server_key_iv: "d987fa0b777d5fba3eb02d8611bbf935"
server_key_version: "encryptionkey@0"
I20260927 00:41:55.942415 7999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:55.956843 7999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:55.959995 7999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:55.960136 7999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:55.961517 7999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:55.961565 7999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:55.961603 7999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:55.961618 7999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:55 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:55.972296 7999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38399
I20260927 00:41:55.973042 8129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38399 every 8 connection(s)
I20260927 00:41:55.973278 7999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:41:55.976423 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.962317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36389 (local address 127.0.98.62:40801)
0927 00:41:55.962479 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:55.962484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:55.963364 (+ 880us) server_negotiation.cc:407] Connection header received
0927 00:41:55.964592 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:55.964595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:55.964657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:55.964750 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:55.966390 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.967286 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.969296 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.969582 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.969706 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:55.972788 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:55.972791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:55.972794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:55.972823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:55.974559 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.975106 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.975109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.975110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.975186 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.975502 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.975504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.975505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.975667 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:55.975740 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:55.976206 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 00:41:55.976306 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 00:41:55.977368 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.962738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.1:36389)
0927 00:41:55.963191 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:55.963243 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:55.964313 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:55.965010 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:55.965017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:55.965448 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:55.966195 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:55.966214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.967446 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.967453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:55.969162 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:55.969176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.969778 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.969786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:55.969961 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:55.970618 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:55.970643 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:55.972606 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:55.974685 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:55.974694 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:55.974710 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:55.975028 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:55.975282 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:55.975285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:55.975286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:55.975418 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:55.975768 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:55.975773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:55.975994 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 00:41:55.976161 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 00:41:55.976415 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":127,"threads_started":1}
I20260927 00:41:55.977900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 7999
I20260927 00:41:55.978070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:41:55.978415 392 external_mini_cluster.cc:1428] Setting key a0c1de73e9e813ad08b37a2b6fb578b2
I20260927 00:41:55.978844 8130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40801
I20260927 00:41:55.978983 8130 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:55.979450 8130 heartbeater.cc:511] Master 127.0.98.62:40801 requested a full tablet report, sending...
I20260927 00:41:55.980840 7926 ts_manager.cc:194] Registered new tserver with Master: a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399)
I20260927 00:41:55.982481 7926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36389
I20260927 00:41:55.988366 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.981153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48815 (local address 127.0.98.62:40801)
0927 00:41:55.981256 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:55.981260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:55.981515 (+ 255us) server_negotiation.cc:407] Connection header received
0927 00:41:55.981624 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:55.981630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:55.981687 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:55.981801 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:55.982298 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.982889 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.983771 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.983990 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.984097 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:55.985283 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:55.985287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:55.985289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:55.985312 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:55.986763 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.987189 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.987194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.987195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.987281 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:55.987557 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:55.987562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:55.987564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:55.987728 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:55.987835 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:55.988097 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 00:41:55.988202 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:41:55.988377 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:55.981165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.1:48815)
0927 00:41:55.981436 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:55.981451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:55.981547 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:55.981928 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:55.981930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:55.982084 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:55.982172 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:55.982177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.983012 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.983020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:55.983642 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:55.983651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:55.984101 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:55.984104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:55.984222 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:55.984733 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:55.984745 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:55.985145 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:55.986888 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:55.986891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:55.986893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:55.987085 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:55.987374 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:55.987379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:55.987381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:55.987467 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:55.987826 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:55.987829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:55.987931 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 00:41:55.988044 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:41:55.988183 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 00:41:56.007318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40801
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:56.114264 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:56.114555 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:56.114632 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:56.118655 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:56.118732 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:56.118801 8139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:56.118880 8139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:41:56.124222 8139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:56.125283 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:56.126279 8139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:56.136175 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:56.136193 8144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:56.136184 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:56.136406 8139 server_base.cc:1034] running on GCE node
I20260927 00:41:56.136745 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:56.137365 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:56.138520 8139 hybrid_clock.cc:648] HybridClock initialized: now 1790469716138471 us; error 67 us; skew 500 ppm
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:56.141206 8139 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:41:56.142019 8139 webserver.cc:460] Webserver started at http://127.0.98.2:35753/ using document root <none> and password file <none>
I20260927 00:41:56.142544 8139 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:56.142604 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:56.142778 8139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:56.144810 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "a99085c2330f43879f4033ecab04c9f4"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "4645b5fa2e65886d34ad167e563c44a1"
server_key_iv: "084146f656c61d2f43fe954bc197fffd"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.145433 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a99085c2330f43879f4033ecab04c9f4"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "4645b5fa2e65886d34ad167e563c44a1"
server_key_iv: "084146f656c61d2f43fe954bc197fffd"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.149184 8139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:41:56.151449 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:56.152575 8139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:41:56.152694 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "a99085c2330f43879f4033ecab04c9f4"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "4645b5fa2e65886d34ad167e563c44a1"
server_key_iv: "084146f656c61d2f43fe954bc197fffd"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.152827 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:56.180328 8139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:56.183101 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:56.183298 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:56.184684 8139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:56.184758 8139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:56.184844 8139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:56.184882 8139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:56.195740 8139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46447
I20260927 00:41:56.196619 8269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46447 every 8 connection(s)
I20260927 00:41:56.196856 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:41:56.198500 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.185575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42453 (local address 127.0.98.62:40801)
0927 00:41:56.185697 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.185701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.186676 (+ 975us) server_negotiation.cc:407] Connection header received
0927 00:41:56.187623 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.187630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.187698 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.187797 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.189397 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.190287 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.191517 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.191795 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.191914 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.194663 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.194669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.194672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.194698 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.196425 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.197055 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.197059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.197061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.197155 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.197542 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.197545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.197547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.197761 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.197860 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.198291 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.198386 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:56.199230 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.186076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.2:42453)
0927 00:41:56.186530 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.186571 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.187369 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.188116 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.188123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.188613 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:56.189231 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.189255 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.190431 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.190434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.191369 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.191378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.191954 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.191958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.192136 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.192703 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:56.192722 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:56.194503 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:56.196566 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.196574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.196587 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.196946 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.197262 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.197265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.197269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.197424 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.197870 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:56.197874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:56.198079 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.198231 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.198457 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260927 00:41:56.200487 8270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40801
I20260927 00:41:56.200632 8270 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:56.201159 8270 heartbeater.cc:511] Master 127.0.98.62:40801 requested a full tablet report, sending...
I20260927 00:41:56.202180 7927 ts_manager.cc:194] Registered new tserver with Master: a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:56.202936 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42453
I20260927 00:41:56.203312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8139
I20260927 00:41:56.203457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:41:56.203810 392 external_mini_cluster.cc:1428] Setting key 6c6f9fd0044fa2471e873c547c166e8b
I20260927 00:41:56.210402 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.202637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41647 (local address 127.0.98.62:40801)
0927 00:41:56.202750 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.202754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.202978 (+ 224us) server_negotiation.cc:407] Connection header received
0927 00:41:56.203170 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.203173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.203226 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.203326 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.203779 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.204427 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.205575 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.205942 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.206068 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.207192 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.207195 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.207197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.207216 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.208629 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.209176 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.209179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.209181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.209264 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.209570 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.209573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.209574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.209780 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.209865 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.210194 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.210292 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:41:56.210556 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.202519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.2:41647)
0927 00:41:56.202890 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.202903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.203010 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.203347 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.203349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.203566 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:56.203667 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.203672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.204593 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.204596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.205448 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.205457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.206057 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.206060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.206125 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.206604 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:56.206616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:56.207059 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:56.208741 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.208747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.208752 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.209052 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.209372 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.209375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.209377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.209473 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.209891 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:56.209896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:56.210017 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.210146 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.210309 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:41:56.230103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40801
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:56.338670 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:56.338907 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:56.338959 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:56.342938 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:56.342984 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:56.343021 8279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 00:41:56.343052 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:41:56.348423 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:56.349470 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:56.350445 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:41:56.359767 8279 server_base.cc:1034] running on GCE node
W20260927 00:41:56.359755 8284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:56.359723 8285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:56.359931 8287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:56.360368 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:56.361009 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:56.362159 8279 hybrid_clock.cc:648] HybridClock initialized: now 1790469716362119 us; error 52 us; skew 500 ppm
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:56.364677 8279 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:41:56.365552 8279 webserver.cc:460] Webserver started at http://127.0.98.3:43867/ using document root <none> and password file <none>
I20260927 00:41:56.366076 8279 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:56.366168 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:56.366380 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:56.368269 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "fbd4042fe71574a2536e8105a5255c44"
server_key_iv: "9ec1a963fb738dcc0dd9ba40962b09a3"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.368818 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "fbd4042fe71574a2536e8105a5255c44"
server_key_iv: "9ec1a963fb738dcc0dd9ba40962b09a3"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.372334 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:56.374590 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:56.375751 8279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:56.375878 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
format_stamp: "Formatted at 2026-09-27 00:41:56 on dist-test-slave-pkws"
server_key: "fbd4042fe71574a2536e8105a5255c44"
server_key_iv: "9ec1a963fb738dcc0dd9ba40962b09a3"
server_key_version: "encryptionkey@0"
I20260927 00:41:56.375985 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:56.393080 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:56.396029 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:56.396253 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:56.397800 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:41:56.397877 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:56.397951 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:41:56.398003 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:56.409986 8279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35371
I20260927 00:41:56.410089 8409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35371 every 8 connection(s)
I20260927 00:41:56.411221 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:41:56.412560 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.398379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47811 (local address 127.0.98.62:40801)
0927 00:41:56.398540 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.398545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.399481 (+ 936us) server_negotiation.cc:407] Connection header received
0927 00:41:56.400838 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.400841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.400932 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.401022 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.402761 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.403298 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.404500 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.404731 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.404840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.408328 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.408331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.408333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.408357 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.410193 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.410815 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.410819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.410820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.410900 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.411423 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.411426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.411428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.411635 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.411731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.412287 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.412398 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:41:56.413333 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.398801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.3:47811)
0927 00:41:56.399287 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.399358 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.400595 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.401224 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.401236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.401811 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:56.402549 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.402573 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.403430 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.403434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.404382 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.404392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.404872 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.404877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.405117 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.405728 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:56.405744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:56.408146 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:56.410312 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.410318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.410331 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.410680 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.410995 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.410998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.411000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.411177 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.411779 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:56.411785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:56.412045 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.412247 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.412539 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":93,"threads_started":1}
I20260927 00:41:56.414438 8410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40801
I20260927 00:41:56.414558 8410 heartbeater.cc:464] Registering TS with master...
I20260927 00:41:56.415073 8410 heartbeater.cc:511] Master 127.0.98.62:40801 requested a full tablet report, sending...
I20260927 00:41:56.415918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8279
I20260927 00:41:56.416079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:41:56.416209 7926 ts_manager.cc:194] Registered new tserver with Master: 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371)
I20260927 00:41:56.416424 392 external_mini_cluster.cc:1428] Setting key d1fe2e05cd3f5e887944ab2f8f0f766e
I20260927 00:41:56.417088 7926 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47811
I20260927 00:41:56.418699 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:41:56.424743 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.416696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.98.3:42977)
0927 00:41:56.416957 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.416977 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.417082 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.417442 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.417445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.417621 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:56.417748 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.417755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.418762 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.418765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.419449 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.419457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.420124 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.420126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.420191 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.420715 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:56.420733 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:56.421204 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:56.423187 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.423193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.423196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.423461 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.423793 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.423796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.423797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.423870 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.424370 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:56.424372 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:56.424454 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.424533 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.424626 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260927 00:41:56.425063 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.416736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42977 (local address 127.0.98.62:40801)
0927 00:41:56.416867 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.416871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.417047 (+ 176us) server_negotiation.cc:407] Connection header received
0927 00:41:56.417194 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.417196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.417242 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.417288 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.417875 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.418609 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.419570 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.420013 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.420144 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.421357 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.421360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.421362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.421384 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.423076 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.423555 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.423558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.423559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.423671 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.423967 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.423971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.423972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.424234 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.424394 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.424626 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.424766 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:41:56.443930 7976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:56.437790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.0.98.62:40801)
0927 00:41:56.437914 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.437917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.437930 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:41:56.437962 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.437964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.437997 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.438065 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.438735 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.439242 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.440109 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.440370 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.440489 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.443822 (+ 3333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52476: BlockingRecv error: recv got EOF from 127.0.0.1:52476 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:56.453406 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.445413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.0.98.62:40801)
0927 00:41:56.445530 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.445533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.445631 (+ 98us) server_negotiation.cc:407] Connection header received
0927 00:41:56.445800 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.445803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.445851 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.445917 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.446808 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.447297 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.448230 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.448422 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.448495 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.450811 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.450814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.450816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.450836 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.452003 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.452362 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.452364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.452365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.452432 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.452644 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.452646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.452647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.452795 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.452952 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.453129 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.453210 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:41:56.456151 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:56.459239 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:56.473600 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.467979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46447 (local address 127.0.0.1:38284)
0927 00:41:56.468730 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.468742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.468845 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.469351 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.469354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.469373 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.469461 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.469509 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470549 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.470555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.471747 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.471756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.473055 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.473058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.473172 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.473210 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.473323 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.473450 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":85,"threads_started":1}
I20260927 00:41:56.473763 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.467310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35371 (local address 127.0.0.1:36908)
0927 00:41:56.468282 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.468295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.468380 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.469184 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.469186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.469204 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.469301 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.469310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470734 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.470737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.472254 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.472260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.473384 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.473388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.473471 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.473497 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.473549 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.473613 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":145,"thread_start_us":85,"threads_started":1}
I20260927 00:41:56.474138 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.467276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38399 (local address 127.0.0.1:34440)
0927 00:41:56.467604 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.467651 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.467766 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.469012 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.469017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.469036 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.469170 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.469176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470734 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.470737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.472249 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.472260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.473564 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.473567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.473641 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.473763 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.473940 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.474004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":205,"thread_start_us":84,"threads_started":1}
I20260927 00:41:56.474383 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.468266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.0.98.3:35371)
0927 00:41:56.468724 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.468734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.468751 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:41:56.468833 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.468838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.469050 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.469200 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.469398 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470605 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.472384 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.473254 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.473944 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.474095 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.474142 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.474256 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":94,"threads_started":1}
I20260927 00:41:56.474593 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.467409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.98.1:38399)
0927 00:41:56.467992 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.468008 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.468401 (+ 393us) server_negotiation.cc:407] Connection header received
0927 00:41:56.468490 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.468495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.468705 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.468934 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.469424 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470605 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.472384 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.473246 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.473363 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.473510 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.474282 (+ 772us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.474363 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":89,"threads_started":1}
I20260927 00:41:56.474728 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.468123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38284 (local address 127.0.98.2:46447)
0927 00:41:56.468597 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.468607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.468840 (+ 233us) server_negotiation.cc:407] Connection header received
0927 00:41:56.468992 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.468997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.469156 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.469279 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.469618 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.470445 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.471866 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.472678 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.472797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.472992 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.473768 (+ 776us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.474593 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":99,"threads_started":1}
I20260927 00:41:56.478178 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 0befc23bffb84f7089310286af144ef1 (DEFAULT_TABLE table=test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.478178 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 0befc23bffb84f7089310286af144ef1 (DEFAULT_TABLE table=test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.478641 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 0befc23bffb84f7089310286af144ef1 (DEFAULT_TABLE table=test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.479245 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0befc23bffb84f7089310286af144ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.479256 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0befc23bffb84f7089310286af144ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.479486 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0befc23bffb84f7089310286af144ef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.485582 8438 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Bootstrap starting.
I20260927 00:41:56.486937 8439 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Bootstrap starting.
I20260927 00:41:56.487040 8440 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Bootstrap starting.
I20260927 00:41:56.487843 8438 tablet_bootstrap.cc:654] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.489068 8440 tablet_bootstrap.cc:654] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.489159 8438 log.cc:824] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:56.489379 8439 tablet_bootstrap.cc:654] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.489787 8440 log.cc:824] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:56.490167 8439 log.cc:824] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:56.491174 8438 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: No bootstrap required, opened a new log
I20260927 00:41:56.491369 8438 ts_tablet_manager.cc:1397] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:41:56.491608 8440 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: No bootstrap required, opened a new log
I20260927 00:41:56.491786 8440 ts_tablet_manager.cc:1397] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:41:56.494033 8439 tablet_bootstrap.cc:492] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: No bootstrap required, opened a new log
I20260927 00:41:56.494220 8439 ts_tablet_manager.cc:1397] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:41:56.495223 8438 raft_consensus.cc:348] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.495359 8440 raft_consensus.cc:348] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.495501 8438 raft_consensus.cc:374] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.495551 8438 raft_consensus.cc:729] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.495558 8440 raft_consensus.cc:374] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.495609 8440 raft_consensus.cc:729] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.495998 8438 consensus_queue.cc:260] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.495998 8440 consensus_queue.cc:260] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.496871 8270 heartbeater.cc:503] Master 127.0.98.62:40801 was elected leader, sending a full tablet report...
I20260927 00:41:56.496922 8410 heartbeater.cc:503] Master 127.0.98.62:40801 was elected leader, sending a full tablet report...
I20260927 00:41:56.496963 8438 ts_tablet_manager.cc:1428] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 00:41:56.497503 8440 ts_tablet_manager.cc:1428] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:41:56.497653 8439 raft_consensus.cc:348] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.497897 8439 raft_consensus.cc:374] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.497956 8439 raft_consensus.cc:729] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.498301 8439 consensus_queue.cc:260] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.498796 8130 heartbeater.cc:503] Master 127.0.98.62:40801 was elected leader, sending a full tablet report...
I20260927 00:41:56.499092 8439 ts_tablet_manager.cc:1428] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:41:56.597893 8446 raft_consensus.cc:482] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:56.598097 8446 raft_consensus.cc:504] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.599190 8446 leader_election.cc:288] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:56.603295 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.599386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35371 (local address 127.0.98.1:56035)
0927 00:41:56.599768 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.599865 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.600024 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.600301 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.600304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.600321 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.600460 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.600485 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.601296 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.601299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.602209 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.602218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.602860 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.602863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.602966 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.602995 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.603084 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.603153 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":160,"threads_started":1}
I20260927 00:41:56.603294 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.599548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56035 (local address 127.0.98.3:35371)
0927 00:41:56.599703 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.599708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.599944 (+ 236us) server_negotiation.cc:407] Connection header received
0927 00:41:56.600136 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.600140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.600198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.600308 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.600577 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.601171 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.602341 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.602733 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.602858 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.602920 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.603099 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.603165 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:41:56.603498 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.600041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42077 (local address 127.0.98.2:46447)
0927 00:41:56.600184 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.600189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.600354 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:41:56.600505 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.600508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.600560 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.600642 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.600940 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.601514 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.602536 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.602926 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.603045 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.603105 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.603308 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.603372 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 00:41:56.603516 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.599899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46447 (local address 127.0.98.1:42077)
0927 00:41:56.600252 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.600263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.600385 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.600653 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.600655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.600671 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.600802 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.600807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.601624 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.601629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.602401 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.602409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.603059 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.603063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.603177 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.603205 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.603292 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.603356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 00:41:56.604359 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0befc23bffb84f7089310286af144ef1" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4" is_pre_election: true
I20260927 00:41:56.604352 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0befc23bffb84f7089310286af144ef1" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" is_pre_election: true
I20260927 00:41:56.604657 8364 raft_consensus.cc:2454] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 0.
I20260927 00:41:56.604673 8224 raft_consensus.cc:2454] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 0.
I20260927 00:41:56.605127 8016 leader_election.cc:302] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6787c475faeb4bbdbe77b98f05ecd73b, a8c7b49d2fca466caff1f946ca06ae07; no voters:
I20260927 00:41:56.605400 8446 raft_consensus.cc:2781] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:56.605490 8446 raft_consensus.cc:482] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:56.605535 8446 raft_consensus.cc:3037] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.606647 8446 raft_consensus.cc:504] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.606967 8446 leader_election.cc:288] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 election: Requested vote from peers 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:56.607259 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0befc23bffb84f7089310286af144ef1" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
I20260927 00:41:56.607352 8364 raft_consensus.cc:3037] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.607345 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0befc23bffb84f7089310286af144ef1" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4"
I20260927 00:41:56.607479 8224 raft_consensus.cc:3037] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.608556 8224 raft_consensus.cc:2454] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 1.
I20260927 00:41:56.608601 8364 raft_consensus.cc:2454] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 1.
I20260927 00:41:56.608878 8016 leader_election.cc:302] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c7b49d2fca466caff1f946ca06ae07, a99085c2330f43879f4033ecab04c9f4; no voters:
I20260927 00:41:56.609076 8446 raft_consensus.cc:2781] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:56.609288 8446 raft_consensus.cc:686] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 LEADER]: Becoming Leader. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Running, Role: LEADER
I20260927 00:41:56.609630 8446 consensus_queue.cc:237] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } }
I20260927 00:41:56.612681 7927 catalog_manager.cc:5495] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:56.658347 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.654853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.98.1:38399)
0927 00:41:56.654987 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.654990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.655005 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:56.655119 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.655121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.655175 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.655263 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:56.655473 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.656027 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.656833 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.657152 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.657258 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.657480 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.657882 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.657968 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.658111 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.658157 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 00:41:56.662223 8411 tablet.cc:2369] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:56.663978 8224 raft_consensus.cc:1261] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Refusing update from remote peer a8c7b49d2fca466caff1f946ca06ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.664001 8364 raft_consensus.cc:1261] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Refusing update from remote peer a8c7b49d2fca466caff1f946ca06ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.664767 8449 consensus_queue.cc:1035] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.664948 8446 consensus_queue.cc:1035] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.673247 8453 mvcc.cc:205] Tried to move back new op lower bound from 7333763959448989696 to 7333763959235817472. Current Snapshot: MvccSnapshot[applied={T|T < 7333763959448989696}]
I20260927 00:41:56.679904 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:56.680064 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:56.682350 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Sending DeleteTablet for 3 replicas of tablet 0befc23bffb84f7089310286af144ef1
I20260927 00:41:56.683054 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet 0befc23bffb84f7089310286af144ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260927 00:41:56.683492 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 0befc23bffb84f7089310286af144ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38284
I20260927 00:41:56.683576 8462 tablet_replica.cc:330] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: stopping tablet replica
I20260927 00:41:56.683857 8462 raft_consensus.cc:2229] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:56.683979 8463 tablet_replica.cc:330] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: stopping tablet replica
I20260927 00:41:56.684087 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0befc23bffb84f7089310286af144ef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36908
I20260927 00:41:56.684284 8462 raft_consensus.cc:2258] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:56.684422 8463 raft_consensus.cc:2229] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:56.684725 8463 raft_consensus.cc:2258] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:56.684849 8464 tablet_replica.cc:330] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: stopping tablet replica
I20260927 00:41:56.685124 8464 raft_consensus.cc:2229] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:56.685108 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:41:56.685426 8464 raft_consensus.cc:2258] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:41:56.685485 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:56.686389 8463 ts_tablet_manager.cc:1905] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.686686 8464 ts_tablet_manager.cc:1905] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.689167 8464 ts_tablet_manager.cc:1918] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:56.689167 8463 ts_tablet_manager.cc:1918] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:56.689266 8464 log.cc:1196] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/0befc23bffb84f7089310286af144ef1
I20260927 00:41:56.689404 8463 log.cc:1196] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/0befc23bffb84f7089310286af144ef1
I20260927 00:41:56.689585 8464 ts_tablet_manager.cc:1939] T 0befc23bffb84f7089310286af144ef1 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting consensus metadata
I20260927 00:41:56.689828 8463 ts_tablet_manager.cc:1939] T 0befc23bffb84f7089310286af144ef1 P a99085c2330f43879f4033ecab04c9f4: Deleting consensus metadata
I20260927 00:41:56.690688 7912 catalog_manager.cc:4841] TS 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371): tablet 0befc23bffb84f7089310286af144ef1 (table test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]) successfully deleted
I20260927 00:41:56.690953 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16dd10bdc6426fabe54475b336e48c (DEFAULT_TABLE table=test-table [id=f10fa8cd53434555994e4f281eb0c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.691239 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16dd10bdc6426fabe54475b336e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.691202 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16dd10bdc6426fabe54475b336e48c (DEFAULT_TABLE table=test-table [id=f10fa8cd53434555994e4f281eb0c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.691365 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 4f16dd10bdc6426fabe54475b336e48c (DEFAULT_TABLE table=test-table [id=f10fa8cd53434555994e4f281eb0c2d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:56.691425 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16dd10bdc6426fabe54475b336e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.691591 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f16dd10bdc6426fabe54475b336e48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.693513 8438 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Bootstrap starting.
I20260927 00:41:56.693748 7912 catalog_manager.cc:4841] TS a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447): tablet 0befc23bffb84f7089310286af144ef1 (table test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]) successfully deleted
I20260927 00:41:56.693892 8439 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Bootstrap starting.
I20260927 00:41:56.694510 8438 tablet_bootstrap.cc:654] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.694820 8439 tablet_bootstrap.cc:654] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.695057 8462 ts_tablet_manager.cc:1905] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.695452 8440 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Bootstrap starting.
I20260927 00:41:56.695537 8438 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: No bootstrap required, opened a new log
I20260927 00:41:56.695600 8438 ts_tablet_manager.cc:1397] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:56.696061 8438 raft_consensus.cc:348] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.696163 8438 raft_consensus.cc:374] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.696197 8438 raft_consensus.cc:729] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.696331 8440 tablet_bootstrap.cc:654] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.696322 8438 consensus_queue.cc:260] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.696588 8438 ts_tablet_manager.cc:1428] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:56.697394 8462 ts_tablet_manager.cc:1918] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:56.697701 8462 log.cc:1196] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/0befc23bffb84f7089310286af144ef1
I20260927 00:41:56.697988 8440 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: No bootstrap required, opened a new log
I20260927 00:41:56.697996 8462 ts_tablet_manager.cc:1939] T 0befc23bffb84f7089310286af144ef1 P a8c7b49d2fca466caff1f946ca06ae07: Deleting consensus metadata
I20260927 00:41:56.698060 8440 ts_tablet_manager.cc:1397] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 00:41:56.698523 8440 raft_consensus.cc:348] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.698629 8440 raft_consensus.cc:374] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.698668 8440 raft_consensus.cc:729] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.698799 8440 consensus_queue.cc:260] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.699062 8440 ts_tablet_manager.cc:1428] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:56.699409 7911 catalog_manager.cc:4841] TS a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399): tablet 0befc23bffb84f7089310286af144ef1 (table test-table [id=137bbe2a3c2e457ea6a95842aff3bd0f]) successfully deleted
I20260927 00:41:56.705547 8439 tablet_bootstrap.cc:492] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: No bootstrap required, opened a new log
I20260927 00:41:56.705623 8439 ts_tablet_manager.cc:1397] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260927 00:41:56.706096 8439 raft_consensus.cc:348] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.706214 8439 raft_consensus.cc:374] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.706272 8439 raft_consensus.cc:729] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.706401 8439 consensus_queue.cc:260] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.706626 8439 ts_tablet_manager.cc:1428] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260927 00:41:56.724965 8131 tablet.cc:2369] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:41:56.754163 8445 raft_consensus.cc:482] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:56.754298 8445 raft_consensus.cc:504] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.755438 8445 leader_election.cc:288] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:56.759447 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.755628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38399 (local address 127.0.98.3:45703)
0927 00:41:56.755798 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.755813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.755889 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.756291 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.756295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.756311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.756422 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.756428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.757521 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.757525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.758344 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.758353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.759023 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.759026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.759131 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.759159 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.759233 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.759299 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 00:41:56.759455 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.755745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45703 (local address 127.0.98.1:38399)
0927 00:41:56.755879 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.755884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.755900 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:56.756041 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.756045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.756096 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.756161 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.756556 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.757397 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.758477 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.758885 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.759006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.759072 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.759266 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.759336 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:41:56.760033 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16dd10bdc6426fabe54475b336e48c" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07" is_pre_election: true
I20260927 00:41:56.760138 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.755659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46447 (local address 127.0.98.3:41749)
0927 00:41:56.756065 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.756079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.756177 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.756593 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.756596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.756620 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:56.756718 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.756723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.757619 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.757622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.758851 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.758862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.759705 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.759709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.759804 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.759825 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.759932 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.760005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":154,"threads_started":1}
I20260927 00:41:56.760169 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.756133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41749 (local address 127.0.98.2:46447)
0927 00:41:56.756266 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.756272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.756306 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:41:56.756358 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.756361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.756411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.756486 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:56.756853 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.757503 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.758986 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.759516 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.759631 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.759708 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.759976 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.760064 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:41:56.760282 8084 raft_consensus.cc:2454] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 0.
I20260927 00:41:56.760620 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16dd10bdc6426fabe54475b336e48c" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4" is_pre_election: true
I20260927 00:41:56.760756 8224 raft_consensus.cc:2454] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 0.
I20260927 00:41:56.760694 8295 leader_election.cc:302] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6787c475faeb4bbdbe77b98f05ecd73b, a8c7b49d2fca466caff1f946ca06ae07; no voters:
I20260927 00:41:56.761023 8445 raft_consensus.cc:2781] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:56.761087 8445 raft_consensus.cc:482] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:56.761139 8445 raft_consensus.cc:3037] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.762322 8445 raft_consensus.cc:504] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.762676 8445 leader_election.cc:288] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 election: Requested vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:56.762984 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16dd10bdc6426fabe54475b336e48c" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4"
I20260927 00:41:56.763047 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f16dd10bdc6426fabe54475b336e48c" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07"
I20260927 00:41:56.763098 8224 raft_consensus.cc:3037] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.763141 8084 raft_consensus.cc:3037] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.764202 8224 raft_consensus.cc:2454] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 1.
I20260927 00:41:56.764201 8084 raft_consensus.cc:2454] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 1.
I20260927 00:41:56.764477 8295 leader_election.cc:302] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6787c475faeb4bbdbe77b98f05ecd73b, a8c7b49d2fca466caff1f946ca06ae07; no voters:
I20260927 00:41:56.764650 8445 raft_consensus.cc:2781] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:56.764931 8445 raft_consensus.cc:686] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 LEADER]: Becoming Leader. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Running, Role: LEADER
I20260927 00:41:56.765182 8445 consensus_queue.cc:237] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.767422 7926 catalog_manager.cc:5495] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } health_report { overall_health: HEALTHY } } }
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:56.808000 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.798467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40801 (local address 127.0.0.1:52486)
0927 00:41:56.798813 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.798827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.798934 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.799203 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.799205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.799570 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:41:56.799657 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.799663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.800753 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.800755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.801510 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.801519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.802107 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.802109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.802170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.802768 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:41:56.802779 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:41:56.804343 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:41:56.806291 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.806299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.806305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.806604 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.806886 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:41:56.806890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:41:56.806892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:41:56.807011 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:41:56.807372 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:41:56.807378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:41:56.807501 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.807627 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.807784 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":140,"threads_started":1}
I20260927 00:41:56.808014 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.798595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.0.98.62:40801)
0927 00:41:56.798742 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.798747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.798917 (+ 170us) server_negotiation.cc:407] Connection header received
0927 00:41:56.799016 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.799019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.799070 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.799167 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.799784 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.800615 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.801629 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.801986 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.802103 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.804492 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:56.804496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:56.804498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:56.804523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:56.806170 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.806703 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.806707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.806708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.806783 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:56.807107 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:56.807109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:56.807111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:56.807262 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:56.807348 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.807675 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.807868 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":59}
I20260927 00:41:56.812163 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52486:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 00:41:56.812959 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:56.818130 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb687344d944761921aa1dffd2efec7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83ac8c3287da43e88fdec42730c3274d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:56.818329 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb687344d944761921aa1dffd2efec7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83ac8c3287da43e88fdec42730c3274d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:56.818557 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb687344d944761921aa1dffd2efec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.818425 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb687344d944761921aa1dffd2efec7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83ac8c3287da43e88fdec42730c3274d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 00:41:56.818663 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb687344d944761921aa1dffd2efec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.818794 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb687344d944761921aa1dffd2efec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:56.821255 8439 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: Bootstrap starting.
I20260927 00:41:56.821998 8438 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: Bootstrap starting.
I20260927 00:41:56.822306 8439 tablet_bootstrap.cc:654] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.822990 8438 tablet_bootstrap.cc:654] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.823204 8440 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: Bootstrap starting.
I20260927 00:41:56.824330 8440 tablet_bootstrap.cc:654] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:56.824873 8438 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: No bootstrap required, opened a new log
I20260927 00:41:56.825009 8438 ts_tablet_manager.cc:1397] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260927 00:41:56.825253 8439 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: No bootstrap required, opened a new log
I20260927 00:41:56.825320 8439 ts_tablet_manager.cc:1397] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 00:41:56.825460 8438 raft_consensus.cc:348] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.825585 8438 raft_consensus.cc:374] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.825621 8438 raft_consensus.cc:729] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.825624 8439 raft_consensus.cc:348] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.825699 8439 raft_consensus.cc:374] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.825758 8439 raft_consensus.cc:729] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.825750 8438 consensus_queue.cc:260] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.825879 8439 consensus_queue.cc:260] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.826169 8445 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.826298 8445 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.826426 8440 tablet_bootstrap.cc:492] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: No bootstrap required, opened a new log
I20260927 00:41:56.826509 8440 ts_tablet_manager.cc:1397] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:41:56.826722 8438 ts_tablet_manager.cc:1428] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 00:41:56.826951 8440 raft_consensus.cc:348] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.827078 8440 raft_consensus.cc:374] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:56.827116 8440 raft_consensus.cc:729] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Initialized, Role: FOLLOWER
I20260927 00:41:56.827239 8440 consensus_queue.cc:260] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.827512 8440 ts_tablet_manager.cc:1428] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:41:56.827458 8444 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.827579 8444 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.827713 8457 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.827841 8457 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.830094 8439 ts_tablet_manager.cc:1428] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260927 00:41:56.846312 8457 raft_consensus.cc:482] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:56.846408 8457 raft_consensus.cc:504] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.846894 8457 leader_election.cc:288] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447), 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371)
I20260927 00:41:56.847244 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb687344d944761921aa1dffd2efec7" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4" is_pre_election: true
I20260927 00:41:56.847278 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb687344d944761921aa1dffd2efec7" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" is_pre_election: true
I20260927 00:41:56.847406 8364 raft_consensus.cc:2454] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 0.
I20260927 00:41:56.847407 8224 raft_consensus.cc:2454] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 0.
I20260927 00:41:56.847676 8016 leader_election.cc:302] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c7b49d2fca466caff1f946ca06ae07, a99085c2330f43879f4033ecab04c9f4; no voters:
I20260927 00:41:56.847860 8457 raft_consensus.cc:2781] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:56.847931 8457 raft_consensus.cc:482] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:56.847981 8457 raft_consensus.cc:3037] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.848851 8457 raft_consensus.cc:504] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.849184 8457 leader_election.cc:288] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 election: Requested vote from peers a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447), 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371)
I20260927 00:41:56.849530 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb687344d944761921aa1dffd2efec7" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
I20260927 00:41:56.849545 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb687344d944761921aa1dffd2efec7" candidate_uuid: "a8c7b49d2fca466caff1f946ca06ae07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4"
I20260927 00:41:56.849641 8364 raft_consensus.cc:3037] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.849665 8224 raft_consensus.cc:3037] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:56.850472 8224 raft_consensus.cc:2454] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 1.
I20260927 00:41:56.850692 8364 raft_consensus.cc:2454] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8c7b49d2fca466caff1f946ca06ae07 in term 1.
I20260927 00:41:56.850764 8016 leader_election.cc:302] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c7b49d2fca466caff1f946ca06ae07, a99085c2330f43879f4033ecab04c9f4; no voters:
I20260927 00:41:56.850941 8457 raft_consensus.cc:2781] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:56.851045 8457 raft_consensus.cc:686] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 LEADER]: Becoming Leader. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Running, Role: LEADER
I20260927 00:41:56.851177 8457 consensus_queue.cc:237] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:56.851814 8446 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: TxnStatusTablet state changed. Reason: New leader a8c7b49d2fca466caff1f946ca06ae07. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.851917 8446 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:56.852197 8481 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 00:41:56.852347 8481 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 00:41:56.852710 7925 catalog_manager.cc:5495] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:56.871637 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.867955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38399 (local address 127.0.0.1:34464)
0927 00:41:56.868093 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.868110 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.868204 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.868471 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.868473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.868481 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:56.868562 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.868567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.869606 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.869609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.870286 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.870295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.870915 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.870918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.870993 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.871034 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.871393 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.871405 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.871461 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.871528 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:41:56.871654 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.868074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.1:38399)
0927 00:41:56.868176 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.868180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.868195 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:56.868316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.868319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.868368 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.868451 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:56.868659 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.869472 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.870398 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.870763 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.870865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.871233 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.871312 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.871381 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.871472 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.871527 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:41:56.874178 8224 raft_consensus.cc:1261] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Refusing update from remote peer a8c7b49d2fca466caff1f946ca06ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.874167 8364 raft_consensus.cc:1261] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Refusing update from remote peer a8c7b49d2fca466caff1f946ca06ae07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.874557 8446 consensus_queue.cc:1035] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.874763 8457 consensus_queue.cc:1035] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.876030 8444 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: TxnStatusTablet state changed. Reason: New leader a8c7b49d2fca466caff1f946ca06ae07. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.876138 8444 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.876879 8445 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: TxnStatusTablet state changed. Reason: New leader a8c7b49d2fca466caff1f946ca06ae07. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.877022 8445 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.878255 8457 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.878365 8457 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:56.878468 8457 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.878541 8457 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a8c7b49d2fca466caff1f946ca06ae07: This TxnStatusTablet replica's current role is: LEADER
I20260927 00:41:56.878615 8452 mvcc.cc:205] Tried to move back new op lower bound from 7333763960312410112 to 7333763960223555584. Current Snapshot: MvccSnapshot[applied={T|T < 7333763960312410112 or (T in {7333763960312410112})}]
I20260927 00:41:56.879520 8454 mvcc.cc:205] Tried to move back new op lower bound from 7333763960312410112 to 7333763960223555584. Current Snapshot: MvccSnapshot[applied={T|T < 7333763960312410112}]
I20260927 00:41:56.880091 8444 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.880180 8444 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P a99085c2330f43879f4033ecab04c9f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.881102 8445 tablet_replica.cc:439] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8c7b49d2fca466caff1f946ca06ae07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } } }
I20260927 00:41:56.881203 8445 tablet_replica.cc:442] T 0cb687344d944761921aa1dffd2efec7 P 6787c475faeb4bbdbe77b98f05ecd73b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 00:41:56.888927 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.883595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.0.98.3:35371)
0927 00:41:56.883694 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.883698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.883818 (+ 120us) server_negotiation.cc:407] Connection header received
0927 00:41:56.884010 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.884013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.884069 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.884202 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:56.884642 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.885574 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.886919 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.887334 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.887440 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.887710 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.888319 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.888439 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.888533 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.888610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:41:56.898658 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.895684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39519 (local address 127.0.98.1:38399)
0927 00:41:56.895797 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.895802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.895816 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:56.895879 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.895882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.895928 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.896018 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:56.896258 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.896753 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.897639 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.897868 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.897975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.898244 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.898317 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.898395 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.898477 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.898529 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:41:56.898658 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.895564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38399 (local address 127.0.98.3:39519)
0927 00:41:56.895701 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.895719 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.895798 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.896024 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.896027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.896034 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:56.896157 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.896162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.896878 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.896881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.897509 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.897517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.897973 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.897976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.898072 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.898109 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.898396 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.898405 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.898470 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.898543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:41:56.904793 8224 raft_consensus.cc:1261] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Refusing update from remote peer 6787c475faeb4bbdbe77b98f05ecd73b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.905014 8084 raft_consensus.cc:1261] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Refusing update from remote peer 6787c475faeb4bbdbe77b98f05ecd73b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:56.905570 8469 consensus_queue.cc:1035] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.905798 8445 consensus_queue.cc:1035] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:56.926681 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.922901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56343 (local address 127.0.98.3:35371)
0927 00:41:56.923028 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.923033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.923284 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:41:56.923436 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.923438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.923491 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.923561 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:56.923832 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.924577 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.925629 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.925864 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.925972 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.926245 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.926323 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.926403 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:56.926496 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 00:41:56.926542 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:41:56.926689 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.922840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35371 (local address 127.0.98.1:56343)
0927 00:41:56.923202 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:56.923213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:56.923330 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:56.923582 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:56.923584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:56.923592 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 00:41:56.923712 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.923718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.924706 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.924709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.925513 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:56.925522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.925972 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.925976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:56.926066 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.926119 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 00:41:56.926421 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 00:41:56.926427 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 00:41:56.926487 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:41:56.926568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260927 00:41:56.953548 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:56.953719 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:56.955977 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Sending DeleteTablet for 3 replicas of tablet 4f16dd10bdc6426fabe54475b336e48c
I20260927 00:41:56.956754 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16dd10bdc6426fabe54475b336e48c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36908
I20260927 00:41:56.956938 8464 tablet_replica.cc:330] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: stopping tablet replica
I20260927 00:41:56.956964 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16dd10bdc6426fabe54475b336e48c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38284
I20260927 00:41:56.957055 8464 raft_consensus.cc:2229] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:56.957126 8463 tablet_replica.cc:330] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: stopping tablet replica
I20260927 00:41:56.957224 8463 raft_consensus.cc:2229] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:56.957301 8464 raft_consensus.cc:2258] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:56.957326 8463 raft_consensus.cc:2258] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:56.958005 8463 ts_tablet_manager.cc:1905] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.958211 8464 ts_tablet_manager.cc:1905] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.960963 8464 ts_tablet_manager.cc:1918] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:56.961032 8464 log.cc:1196] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/4f16dd10bdc6426fabe54475b336e48c
I20260927 00:41:56.961323 8464 ts_tablet_manager.cc:1939] T 4f16dd10bdc6426fabe54475b336e48c P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting consensus metadata
I20260927 00:41:56.962026 7912 catalog_manager.cc:4841] TS 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371): tablet 4f16dd10bdc6426fabe54475b336e48c (table test-table [id=f10fa8cd53434555994e4f281eb0c2d0]) successfully deleted
I20260927 00:41:56.962519 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f16dd10bdc6426fabe54475b336e48c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260927 00:41:56.962685 8462 tablet_replica.cc:330] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: stopping tablet replica
I20260927 00:41:56.962778 8463 ts_tablet_manager.cc:1918] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:56.962790 8462 raft_consensus.cc:2229] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:56.962854 8463 log.cc:1196] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/4f16dd10bdc6426fabe54475b336e48c
I20260927 00:41:56.962888 8462 raft_consensus.cc:2258] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:56.963086 8463 ts_tablet_manager.cc:1939] T 4f16dd10bdc6426fabe54475b336e48c P a99085c2330f43879f4033ecab04c9f4: Deleting consensus metadata
I20260927 00:41:56.963687 7912 catalog_manager.cc:4841] TS a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447): tablet 4f16dd10bdc6426fabe54475b336e48c (table test-table [id=f10fa8cd53434555994e4f281eb0c2d0]) successfully deleted
I20260927 00:41:56.966583 8462 ts_tablet_manager.cc:1905] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:56.969192 8462 ts_tablet_manager.cc:1918] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:56.969261 8462 log.cc:1196] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/4f16dd10bdc6426fabe54475b336e48c
I20260927 00:41:56.969501 8462 ts_tablet_manager.cc:1939] T 4f16dd10bdc6426fabe54475b336e48c P a8c7b49d2fca466caff1f946ca06ae07: Deleting consensus metadata
I20260927 00:41:56.970474 7911 catalog_manager.cc:4841] TS a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399): tablet 4f16dd10bdc6426fabe54475b336e48c (table test-table [id=f10fa8cd53434555994e4f281eb0c2d0]) successfully deleted
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:41:56 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:41:56.994824 7976 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:41:56.986256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.0.98.62:40801)
0927 00:41:56.986391 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.986396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.986972 (+ 576us) server_negotiation.cc:407] Connection header received
0927 00:41:56.987180 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.987183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.987236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.987328 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.988352 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.989259 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.990542 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.990880 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:56.990963 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:56.994704 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52494: BlockingRecv error: recv got EOF from 127.0.0.1:52494 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 00:41:57 dist-test-slave-pkws krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 00:41:57.007808 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:56.997535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.98.62:40801)
0927 00:41:56.997636 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:56.997640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:56.998143 (+ 503us) server_negotiation.cc:407] Connection header received
0927 00:41:56.998307 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:56.998310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:56.998351 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:56.998397 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:56.999207 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:56.999961 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.001170 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.001505 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.001605 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.004363 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:57.004367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:57.004370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:57.004394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:57.006080 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:57.006579 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:57.006583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:57.006585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:57.006667 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:57.006957 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:57.006961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:57.006963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:57.007171 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:57.007299 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.007547 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.007653 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:41:57.010613 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:41:57.011059 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:57.016047 8064 tablet_service.cc:1467] Processing CreateTablet for tablet d3de9dc0c0d2466e9ba26a363494775b (DEFAULT_TABLE table=test-table [id=12106ebe46984f0583b2202b474bb22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.016332 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3de9dc0c0d2466e9ba26a363494775b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.016342 8204 tablet_service.cc:1467] Processing CreateTablet for tablet d3de9dc0c0d2466e9ba26a363494775b (DEFAULT_TABLE table=test-table [id=12106ebe46984f0583b2202b474bb22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.016597 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3de9dc0c0d2466e9ba26a363494775b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.017143 8344 tablet_service.cc:1467] Processing CreateTablet for tablet d3de9dc0c0d2466e9ba26a363494775b (DEFAULT_TABLE table=test-table [id=12106ebe46984f0583b2202b474bb22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.017407 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3de9dc0c0d2466e9ba26a363494775b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.018898 8440 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Bootstrap starting.
I20260927 00:41:57.019755 8440 tablet_bootstrap.cc:654] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.019812 8438 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Bootstrap starting.
I20260927 00:41:57.020640 8438 tablet_bootstrap.cc:654] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.022387 8438 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: No bootstrap required, opened a new log
I20260927 00:41:57.022465 8438 ts_tablet_manager.cc:1397] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 00:41:57.022867 8438 raft_consensus.cc:348] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.022962 8438 raft_consensus.cc:374] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.022990 8438 raft_consensus.cc:729] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.023108 8438 consensus_queue.cc:260] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.023166 8439 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Bootstrap starting.
I20260927 00:41:57.023319 8438 ts_tablet_manager.cc:1428] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:57.023975 8440 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: No bootstrap required, opened a new log
I20260927 00:41:57.023984 8439 tablet_bootstrap.cc:654] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.024056 8440 ts_tablet_manager.cc:1397] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 00:41:57.024559 8440 raft_consensus.cc:348] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.024657 8440 raft_consensus.cc:374] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.024685 8440 raft_consensus.cc:729] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.024788 8440 consensus_queue.cc:260] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.025043 8440 ts_tablet_manager.cc:1428] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:57.027500 8444 raft_consensus.cc:482] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:57.027581 8444 raft_consensus.cc:504] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.028671 8444 leader_election.cc:288] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371)
I20260927 00:41:57.029668 8439 tablet_bootstrap.cc:492] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: No bootstrap required, opened a new log
I20260927 00:41:57.029781 8439 ts_tablet_manager.cc:1397] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260927 00:41:57.030179 8439 raft_consensus.cc:348] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.030292 8439 raft_consensus.cc:374] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.030336 8439 raft_consensus.cc:729] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.030462 8439 consensus_queue.cc:260] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.030743 8439 ts_tablet_manager.cc:1428] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:57.035142 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.028838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38399 (local address 127.0.98.2:35773)
0927 00:41:57.029529 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:57.029545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:57.029640 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:57.031235 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:57.031238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:57.031256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:57.031359 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:57.031364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.032218 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.032223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:57.033969 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:57.033977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.034731 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.034735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:57.034832 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.034856 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:41:57.034932 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:41:57.034996 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":105,"threads_started":1}
I20260927 00:41:57.035159 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.030683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35773 (local address 127.0.98.1:38399)
0927 00:41:57.030933 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:57.030937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:57.030952 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:41:57.030996 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:57.030999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:57.031033 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:57.031130 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:57.031468 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.032093 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.034092 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.034492 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.034613 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.034674 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.034971 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.035047 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260927 00:41:57.035658 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3de9dc0c0d2466e9ba26a363494775b" candidate_uuid: "a99085c2330f43879f4033ecab04c9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07" is_pre_election: true
I20260927 00:41:57.035780 8084 raft_consensus.cc:2454] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99085c2330f43879f4033ecab04c9f4 in term 0.
I20260927 00:41:57.035858 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.030551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57361 (local address 127.0.98.3:35371)
0927 00:41:57.030676 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:57.030681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:57.030695 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:41:57.030737 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:57.030741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:57.030788 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:57.030875 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:41:57.031971 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.032731 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.034579 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.035165 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.035277 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.035330 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.035671 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.035741 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 00:41:57.036120 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.029190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35371 (local address 127.0.98.2:57361)
0927 00:41:57.030136 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0927 00:41:57.030150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:41:57.030232 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:41:57.031129 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:41:57.031132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:41:57.031151 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:41:57.031837 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:57.031845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.033204 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.033207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:57.034439 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:41:57.034449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.035447 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.035450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:41:57.035518 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.035559 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:41:57.035638 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 00:41:57.035708 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":99,"threads_started":1}
I20260927 00:41:57.036206 8155 leader_election.cc:302] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c7b49d2fca466caff1f946ca06ae07, a99085c2330f43879f4033ecab04c9f4; no voters:
I20260927 00:41:57.036454 8444 raft_consensus.cc:2781] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:57.036517 8444 raft_consensus.cc:482] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:57.036563 8444 raft_consensus.cc:3037] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.036550 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3de9dc0c0d2466e9ba26a363494775b" candidate_uuid: "a99085c2330f43879f4033ecab04c9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" is_pre_election: true
I20260927 00:41:57.036679 8364 raft_consensus.cc:2454] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99085c2330f43879f4033ecab04c9f4 in term 0.
I20260927 00:41:57.037585 8444 raft_consensus.cc:504] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.037950 8444 leader_election.cc:288] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [CANDIDATE]: Term 1 election: Requested vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371)
I20260927 00:41:57.038189 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3de9dc0c0d2466e9ba26a363494775b" candidate_uuid: "a99085c2330f43879f4033ecab04c9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07"
I20260927 00:41:57.038304 8084 raft_consensus.cc:3037] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.038305 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3de9dc0c0d2466e9ba26a363494775b" candidate_uuid: "a99085c2330f43879f4033ecab04c9f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6787c475faeb4bbdbe77b98f05ecd73b"
I20260927 00:41:57.038414 8364 raft_consensus.cc:3037] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.039261 8084 raft_consensus.cc:2454] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99085c2330f43879f4033ecab04c9f4 in term 1.
I20260927 00:41:57.039443 8364 raft_consensus.cc:2454] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99085c2330f43879f4033ecab04c9f4 in term 1.
I20260927 00:41:57.039549 8155 leader_election.cc:302] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8c7b49d2fca466caff1f946ca06ae07, a99085c2330f43879f4033ecab04c9f4; no voters:
I20260927 00:41:57.039750 8444 raft_consensus.cc:2781] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:57.039952 8444 raft_consensus.cc:686] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 1 LEADER]: Becoming Leader. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Running, Role: LEADER
I20260927 00:41:57.040184 8444 consensus_queue.cc:237] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.042421 7927 catalog_manager.cc:5495] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a99085c2330f43879f4033ecab04c9f4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a99085c2330f43879f4033ecab04c9f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } health_report { overall_health: UNKNOWN } } }
I20260927 00:41:57.055497 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.052006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.98.2:46447)
0927 00:41:57.052108 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:57.052114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:57.052169 (+ 55us) server_negotiation.cc:407] Connection header received
0927 00:41:57.052330 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:57.052333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:57.052393 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:57.052501 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:57.052733 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.053291 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.054124 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.054352 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.054457 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.054719 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:57.055145 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:57.055270 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.055331 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.055376 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:41:57.057899 8084 raft_consensus.cc:1261] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Refusing update from remote peer a99085c2330f43879f4033ecab04c9f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:57.057914 8364 raft_consensus.cc:1261] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Refusing update from remote peer a99085c2330f43879f4033ecab04c9f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:57.058331 8444 consensus_queue.cc:1035] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:57.058552 8512 consensus_queue.cc:1035] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:57.070158 7927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:57.070317 7927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:57.072458 7927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Sending DeleteTablet for 3 replicas of tablet d3de9dc0c0d2466e9ba26a363494775b
I20260927 00:41:57.073038 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet d3de9dc0c0d2466e9ba26a363494775b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260927 00:41:57.073202 8462 tablet_replica.cc:330] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: stopping tablet replica
I20260927 00:41:57.073225 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet d3de9dc0c0d2466e9ba26a363494775b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38284
I20260927 00:41:57.073310 8462 raft_consensus.cc:2229] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:57.073376 8463 tablet_replica.cc:330] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: stopping tablet replica
I20260927 00:41:57.073419 8462 raft_consensus.cc:2258] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.073472 8463 raft_consensus.cc:2229] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:57.073714 8463 raft_consensus.cc:2258] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.074188 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet d3de9dc0c0d2466e9ba26a363494775b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36908
I20260927 00:41:57.074337 8464 tablet_replica.cc:330] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: stopping tablet replica
I20260927 00:41:57.074420 8462 ts_tablet_manager.cc:1905] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:57.074434 8464 raft_consensus.cc:2229] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:57.074544 8464 raft_consensus.cc:2258] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.075282 8464 ts_tablet_manager.cc:1905] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:57.076963 8462 ts_tablet_manager.cc:1918] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:57.077031 8462 log.cc:1196] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/d3de9dc0c0d2466e9ba26a363494775b
I20260927 00:41:57.077281 8462 ts_tablet_manager.cc:1939] T d3de9dc0c0d2466e9ba26a363494775b P a8c7b49d2fca466caff1f946ca06ae07: Deleting consensus metadata
I20260927 00:41:57.077669 8464 ts_tablet_manager.cc:1918] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:57.077737 8464 log.cc:1196] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/d3de9dc0c0d2466e9ba26a363494775b
I20260927 00:41:57.077998 8464 ts_tablet_manager.cc:1939] T d3de9dc0c0d2466e9ba26a363494775b P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting consensus metadata
I20260927 00:41:57.078181 7911 catalog_manager.cc:4841] TS a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399): tablet d3de9dc0c0d2466e9ba26a363494775b (table test-table [id=12106ebe46984f0583b2202b474bb22b]) successfully deleted
I20260927 00:41:57.078586 7912 catalog_manager.cc:4841] TS 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371): tablet d3de9dc0c0d2466e9ba26a363494775b (table test-table [id=12106ebe46984f0583b2202b474bb22b]) successfully deleted
I20260927 00:41:57.079105 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 00:41:57.079270 8463 ts_tablet_manager.cc:1905] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:41:57.079523 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:41:57.081635 8463 ts_tablet_manager.cc:1918] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:41:57.081712 8463 log.cc:1196] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/d3de9dc0c0d2466e9ba26a363494775b
I20260927 00:41:57.081987 8463 ts_tablet_manager.cc:1939] T d3de9dc0c0d2466e9ba26a363494775b P a99085c2330f43879f4033ecab04c9f4: Deleting consensus metadata
I20260927 00:41:57.082546 7912 catalog_manager.cc:4841] TS a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447): tablet d3de9dc0c0d2466e9ba26a363494775b (table test-table [id=12106ebe46984f0583b2202b474bb22b]) successfully deleted
I20260927 00:41:57.085804 8064 tablet_service.cc:1467] Processing CreateTablet for tablet c35b17af02594f7ba250d7d19d197702 (DEFAULT_TABLE table=test-table [id=34638ba499f94c6d8a1c231028a03d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.086006 8204 tablet_service.cc:1467] Processing CreateTablet for tablet c35b17af02594f7ba250d7d19d197702 (DEFAULT_TABLE table=test-table [id=34638ba499f94c6d8a1c231028a03d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.086079 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b17af02594f7ba250d7d19d197702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.086213 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b17af02594f7ba250d7d19d197702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.088219 8440 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Bootstrap starting.
I20260927 00:41:57.088327 8439 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Bootstrap starting.
I20260927 00:41:57.089098 8440 tablet_bootstrap.cc:654] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.089165 8439 tablet_bootstrap.cc:654] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.089990 8344 tablet_service.cc:1467] Processing CreateTablet for tablet c35b17af02594f7ba250d7d19d197702 (DEFAULT_TABLE table=test-table [id=34638ba499f94c6d8a1c231028a03d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:41:57.090219 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35b17af02594f7ba250d7d19d197702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:57.091467 8439 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: No bootstrap required, opened a new log
I20260927 00:41:57.091560 8439 ts_tablet_manager.cc:1397] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 00:41:57.091948 8439 raft_consensus.cc:348] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.092051 8439 raft_consensus.cc:374] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.092083 8439 raft_consensus.cc:729] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c7b49d2fca466caff1f946ca06ae07, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.092197 8439 consensus_queue.cc:260] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.092293 8438 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Bootstrap starting.
I20260927 00:41:57.092447 8439 ts_tablet_manager.cc:1428] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:57.093128 8438 tablet_bootstrap.cc:654] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:57.094609 8438 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: No bootstrap required, opened a new log
I20260927 00:41:57.094687 8438 ts_tablet_manager.cc:1397] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 00:41:57.095074 8438 raft_consensus.cc:348] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.095180 8438 raft_consensus.cc:374] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.095217 8438 raft_consensus.cc:729] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.095328 8438 consensus_queue.cc:260] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.095582 8438 ts_tablet_manager.cc:1428] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 00:41:57.096381 8440 tablet_bootstrap.cc:492] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: No bootstrap required, opened a new log
I20260927 00:41:57.096458 8440 ts_tablet_manager.cc:1397] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260927 00:41:57.096858 8440 raft_consensus.cc:348] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.097013 8440 raft_consensus.cc:374] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:57.097045 8440 raft_consensus.cc:729] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99085c2330f43879f4033ecab04c9f4, State: Initialized, Role: FOLLOWER
I20260927 00:41:57.097147 8440 consensus_queue.cc:260] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.097376 8440 ts_tablet_manager.cc:1428] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 00:41:57.124154 8469 raft_consensus.cc:482] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:41:57.124276 8469 raft_consensus.cc:504] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.124657 8469 leader_election.cc:288] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:57.125054 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b17af02594f7ba250d7d19d197702" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07" is_pre_election: true
I20260927 00:41:57.125196 8084 raft_consensus.cc:2454] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 0.
I20260927 00:41:57.125192 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b17af02594f7ba250d7d19d197702" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4" is_pre_election: true
I20260927 00:41:57.125334 8224 raft_consensus.cc:2454] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 0.
I20260927 00:41:57.125483 8295 leader_election.cc:302] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6787c475faeb4bbdbe77b98f05ecd73b, a8c7b49d2fca466caff1f946ca06ae07; no voters:
I20260927 00:41:57.125631 8469 raft_consensus.cc:2781] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:41:57.125671 8469 raft_consensus.cc:482] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:41:57.125706 8469 raft_consensus.cc:3037] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.126883 8469 raft_consensus.cc:504] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.127206 8469 leader_election.cc:288] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 election: Requested vote from peers a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399), a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447)
I20260927 00:41:57.127494 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b17af02594f7ba250d7d19d197702" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8c7b49d2fca466caff1f946ca06ae07"
I20260927 00:41:57.127540 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35b17af02594f7ba250d7d19d197702" candidate_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99085c2330f43879f4033ecab04c9f4"
I20260927 00:41:57.127590 8084 raft_consensus.cc:3037] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.127636 8224 raft_consensus.cc:3037] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:57.128700 8084 raft_consensus.cc:2454] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 1.
I20260927 00:41:57.128714 8224 raft_consensus.cc:2454] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6787c475faeb4bbdbe77b98f05ecd73b in term 1.
I20260927 00:41:57.129017 8295 leader_election.cc:302] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6787c475faeb4bbdbe77b98f05ecd73b, a8c7b49d2fca466caff1f946ca06ae07; no voters:
I20260927 00:41:57.129215 8469 raft_consensus.cc:2781] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:57.129308 8469 raft_consensus.cc:686] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 LEADER]: Becoming Leader. State: Replica: 6787c475faeb4bbdbe77b98f05ecd73b, State: Running, Role: LEADER
I20260927 00:41:57.129431 8469 consensus_queue.cc:237] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } }
I20260927 00:41:57.131084 7925 catalog_manager.cc:5495] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6787c475faeb4bbdbe77b98f05ecd73b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35371 } health_report { overall_health: HEALTHY } } }
I20260927 00:41:57.155822 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:57.152341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.0.98.3:35371)
0927 00:41:57.152438 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:57.152443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:57.152459 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:41:57.152512 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:57.152515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:57.152560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:57.152636 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:41:57.152901 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.153633 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.154774 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:57.154976 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:57.155054 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:57.155312 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:41:57.155426 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:41:57.155500 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:57.155615 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:41:57.155658 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:41:57.162010 8224 raft_consensus.cc:1261] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Refusing update from remote peer 6787c475faeb4bbdbe77b98f05ecd73b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:57.162015 8084 raft_consensus.cc:1261] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Refusing update from remote peer 6787c475faeb4bbdbe77b98f05ecd73b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:41:57.162439 8445 consensus_queue.cc:1035] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99085c2330f43879f4033ecab04c9f4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:57.162650 8469 consensus_queue.cc:1035] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8c7b49d2fca466caff1f946ca06ae07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:41:57.193702 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:57.193867 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:41:57.195966 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b692428d1d84c9cb784d19c26a6d8a0: Sending DeleteTablet for 3 replicas of tablet c35b17af02594f7ba250d7d19d197702
I20260927 00:41:57.196439 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet c35b17af02594f7ba250d7d19d197702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34440
I20260927 00:41:57.196452 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet c35b17af02594f7ba250d7d19d197702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38284
I20260927 00:41:57.196635 8462 tablet_replica.cc:330] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: stopping tablet replica
I20260927 00:41:57.196762 8462 raft_consensus.cc:2229] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:57.196866 8462 raft_consensus.cc:2258] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.197048 8463 tablet_replica.cc:330] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: stopping tablet replica
I20260927 00:41:57.197149 8463 raft_consensus.cc:2229] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:41:57.197249 8463 raft_consensus.cc:2258] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.197837 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet c35b17af02594f7ba250d7d19d197702 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:41:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36908
I20260927 00:41:57.198019 8464 tablet_replica.cc:330] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: stopping tablet replica
I20260927 00:41:57.198033 8462 ts_tablet_manager.cc:1905] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:57.198107 8464 raft_consensus.cc:2229] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:41:57.198259 8464 raft_consensus.cc:2258] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:41:57.198983 8464 ts_tablet_manager.cc:1905] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:57.200400 8462 ts_tablet_manager.cc:1918] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:57.200464 8462 log.cc:1196] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/c35b17af02594f7ba250d7d19d197702
I20260927 00:41:57.200711 8462 ts_tablet_manager.cc:1939] T c35b17af02594f7ba250d7d19d197702 P a8c7b49d2fca466caff1f946ca06ae07: Deleting consensus metadata
I20260927 00:41:57.201310 7911 catalog_manager.cc:4841] TS a8c7b49d2fca466caff1f946ca06ae07 (127.0.98.1:38399): tablet c35b17af02594f7ba250d7d19d197702 (table test-table [id=34638ba499f94c6d8a1c231028a03d8c]) successfully deleted
I20260927 00:41:57.201645 8464 ts_tablet_manager.cc:1918] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:57.201709 8464 log.cc:1196] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/c35b17af02594f7ba250d7d19d197702
I20260927 00:41:57.201979 8464 ts_tablet_manager.cc:1939] T c35b17af02594f7ba250d7d19d197702 P 6787c475faeb4bbdbe77b98f05ecd73b: Deleting consensus metadata
I20260927 00:41:57.202152 8463 ts_tablet_manager.cc:1905] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:41:57.202554 7912 catalog_manager.cc:4841] TS 6787c475faeb4bbdbe77b98f05ecd73b (127.0.98.3:35371): tablet c35b17af02594f7ba250d7d19d197702 (table test-table [id=34638ba499f94c6d8a1c231028a03d8c]) successfully deleted
I20260927 00:41:57.204288 8463 ts_tablet_manager.cc:1918] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 00:41:57.204355 8463 log.cc:1196] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/c35b17af02594f7ba250d7d19d197702
I20260927 00:41:57.204531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7999
I20260927 00:41:57.204592 8463 ts_tablet_manager.cc:1939] T c35b17af02594f7ba250d7d19d197702 P a99085c2330f43879f4033ecab04c9f4: Deleting consensus metadata
I20260927 00:41:57.206466 7912 catalog_manager.cc:4841] TS a99085c2330f43879f4033ecab04c9f4 (127.0.98.2:46447): tablet c35b17af02594f7ba250d7d19d197702 (table test-table [id=34638ba499f94c6d8a1c231028a03d8c]) successfully deleted
I20260927 00:41:57.215015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8139
I20260927 00:41:57.221055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8279
I20260927 00:41:57.228035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 7895
2026-09-27T00:41:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2866 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:57 dist-test-slave-pkws krb5kdc[8527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:57 dist-test-slave-pkws krb5kdc[8527](info): set up 2 sockets
Sep 27 00:41:57 dist-test-slave-pkws krb5kdc[8527](info): commencing operation
krb5kdc: starting...
W20260927 00:41:58.273646 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:58 dist-test-slave-pkws krb5kdc[8527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:58.399773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34561
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:58.509541 8540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:58.509768 8540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:58.509804 8540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:58.514222 8540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:58.514273 8540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:58.514293 8540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:58.514310 8540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:58.514325 8540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:58.520021 8540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:58.520980 8540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:58.521965 8540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:58.529922 8548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.529934 8545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:58.529937 8546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:58.530258 8540 server_base.cc:1034] running on GCE node
I20260927 00:41:58.530772 8540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:58.531742 8540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:58.532938 8540 hybrid_clock.cc:648] HybridClock initialized: now 1790469718532911 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T00:41:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1309 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:41:58 dist-test-slave-pkws krb5kdc[8553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:41:58 dist-test-slave-pkws krb5kdc[8553](info): set up 2 sockets
Sep 27 00:41:58 dist-test-slave-pkws krb5kdc[8553](info): commencing operation
krb5kdc: starting...
W20260927 00:41:59.573663 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:41:59 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:41:59.693174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:41:59.799009 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:41:59.799243 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:41:59.799280 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:41:59.803124 8566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:41:59.803170 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:41:59.803189 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:41:59.803205 8566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:41:59.803220 8566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:41:59.808197 8566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:41:59.809088 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:41:59.809985 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:41:59.818711 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:41:59.818683 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:59.818746 8566 server_base.cc:1034] running on GCE node
W20260927 00:41:59.818702 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:41:59.819475 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:41:59.820451 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:41:59.821630 8566 hybrid_clock.cc:648] HybridClock initialized: now 1790469719821615 us; error 24 us; skew 500 ppm
Sep 27 00:41:59 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:41:59.828876 8566 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:41:59.830083 8566 webserver.cc:460] Webserver started at http://127.0.98.62:34447/ using document root <none> and password file <none>
I20260927 00:41:59.830751 8566 fs_manager.cc:359] Metadata directory not provided
I20260927 00:41:59.830838 8566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:41:59.831075 8566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:41:59.833501 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "c8811f80df7847d58b76cb0da928f3ce"
format_stamp: "Formatted at 2026-09-27 00:41:59 on dist-test-slave-pkws"
server_key: "af8e667663dd8f44024f1842735e518b"
server_key_iv: "6ca9f29b89e5d7c0fbf17cb8451371ed"
server_key_version: "encryptionkey@0"
I20260927 00:41:59.833998 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8811f80df7847d58b76cb0da928f3ce"
format_stamp: "Formatted at 2026-09-27 00:41:59 on dist-test-slave-pkws"
server_key: "af8e667663dd8f44024f1842735e518b"
server_key_iv: "6ca9f29b89e5d7c0fbf17cb8451371ed"
server_key_version: "encryptionkey@0"
I20260927 00:41:59.837486 8566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:41:59.839696 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:41:59.840868 8566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:41:59.841025 8566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "c8811f80df7847d58b76cb0da928f3ce"
format_stamp: "Formatted at 2026-09-27 00:41:59 on dist-test-slave-pkws"
server_key: "af8e667663dd8f44024f1842735e518b"
server_key_iv: "6ca9f29b89e5d7c0fbf17cb8451371ed"
server_key_version: "encryptionkey@0"
I20260927 00:41:59.841107 8566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:41:59.860190 8566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:41:59.864364 8566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:41:59.864573 8566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:41:59.873878 8566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42953
I20260927 00:41:59.873975 8633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42953 every 8 connection(s)
I20260927 00:41:59.875164 8566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:41:59.878079 8634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:41:59.879482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8566
I20260927 00:41:59.879623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:41:59.879940 392 external_mini_cluster.cc:1428] Setting key 85a44c5c49f7a56e2865326859747ba1
I20260927 00:41:59.884662 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Bootstrap starting.
I20260927 00:41:59.887035 8634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Neither blocks nor log segments found. Creating new log.
I20260927 00:41:59.887885 8634 log.cc:824] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 00:41:59.890153 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: No bootstrap required, opened a new log
Sep 27 00:41:59 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:41:59.893988 8634 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:41:59.894251 8634 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:41:59.894315 8634 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8811f80df7847d58b76cb0da928f3ce, State: Initialized, Role: FOLLOWER
I20260927 00:41:59.894845 8634 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:41:59.894966 8634 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:41:59.895005 8634 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:41:59.895066 8634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:41:59.896091 8634 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:41:59.896448 8634 leader_election.cc:302] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8811f80df7847d58b76cb0da928f3ce; no voters:
I20260927 00:41:59.896809 8634 leader_election.cc:288] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:41:59.896999 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:41:59.897310 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 LEADER]: Becoming Leader. State: Replica: c8811f80df7847d58b76cb0da928f3ce, State: Running, Role: LEADER
I20260927 00:41:59.897673 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:41:59.898075 8634 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:41:59.898535 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:41:59.881390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.0.98.62:42953)
0927 00:41:59.881880 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 00:41:59.881901 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:41:59.881948 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:41:59.882940 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:41:59.882966 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:41:59.883403 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:41:59.883774 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:41:59.884636 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:59.886356 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:59.887671 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:41:59.888562 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:41:59.888753 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:41:59.891869 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:41:59.891875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:41:59.891891 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:41:59.891921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:41:59.894669 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:59.895205 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:59.895213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:59.895222 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:59.895365 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:41:59.895688 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:41:59.895691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:41:59.895693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:41:59.896216 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:41:59.897165 (+ 949us) server_negotiation.cc:1035] Waiting for connection context
0927 00:41:59.897318 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 00:41:59.897604 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260927 00:41:59.899266 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8811f80df7847d58b76cb0da928f3ce. Latest consensus state: current_term: 1 leader_uuid: "c8811f80df7847d58b76cb0da928f3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260927 00:41:59.899477 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: This master's current role is: LEADER
I20260927 00:41:59.899693 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8811f80df7847d58b76cb0da928f3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260927 00:41:59.899780 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: This master's current role is: LEADER
W20260927 00:41:59.903102 8654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:41:59.903178 8654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:41:59.903234 8650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:41:59.904022 8650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:41:59.909888 8650 catalog_manager.cc:1347] Generated new cluster ID: 99a2d0e721584968b0ada13566f49a9f
I20260927 00:41:59.909948 8650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:41:59.920416 8650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:41:59.921663 8650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:41:59.936159 8650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Generated new TSK 0
I20260927 00:41:59.936916 8650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:41:59.967492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:00.075336 8660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:00.075594 8660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:00.075664 8660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:00.079669 8660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:00.079751 8660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:00.079843 8660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:00.084831 8660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:00.085816 8660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:00.086787 8660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:00.096025 8668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.096136 8666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.096101 8660 server_base.cc:1034] running on GCE node
W20260927 00:42:00.096050 8665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.096660 8660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:00.097311 8660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:00.098505 8660 hybrid_clock.cc:648] HybridClock initialized: now 1790469720098485 us; error 43 us; skew 500 ppm
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:00.101364 8660 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:42:00.102123 8660 webserver.cc:460] Webserver started at http://127.0.98.1:34587/ using document root <none> and password file <none>
I20260927 00:42:00.102667 8660 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:00.102730 8660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:00.102887 8660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:00.104753 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "f2239511d4f247a3a4265c6d524c242e"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "885e17c98892ad26a15d34ee6f1bbbb9"
server_key_iv: "625e537fb6a8e9fe514535f3544941ad"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.105302 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f2239511d4f247a3a4265c6d524c242e"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "885e17c98892ad26a15d34ee6f1bbbb9"
server_key_iv: "625e537fb6a8e9fe514535f3544941ad"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.108690 8660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:00.111106 8675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.112337 8660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:42:00.112475 8660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f2239511d4f247a3a4265c6d524c242e"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "885e17c98892ad26a15d34ee6f1bbbb9"
server_key_iv: "625e537fb6a8e9fe514535f3544941ad"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.112586 8660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:00.129240 8660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:00.132360 8660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:00.132562 8660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:00.133179 8660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:00.133946 8660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:00.134016 8660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.134083 8660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:00.134117 8660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.142271 8660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39607
I20260927 00:42:00.142318 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39607 every 8 connection(s)
I20260927 00:42:00.143190 8660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:00.153383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8660
I20260927 00:42:00.153539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:00.153891 392 external_mini_cluster.cc:1428] Setting key a2743de3a2b8870c8b771ec445319193
I20260927 00:42:00.157204 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.145112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33875 (local address 127.0.98.62:42953)
0927 00:42:00.145240 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:00.145244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:00.146038 (+ 794us) server_negotiation.cc:407] Connection header received
0927 00:42:00.147428 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:00.147431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:00.147488 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:00.147566 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:00.148684 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.149281 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.150431 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.150658 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.150767 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.153090 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:00.153093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:00.153095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:00.153119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:00.154677 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.155191 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.155199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.155201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.155301 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.155677 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.155682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.155684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.155928 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:00.156038 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:00.156722 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 00:42:00.156933 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:42:00.158048 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.145478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:33875)
0927 00:42:00.145890 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:00.145931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:00.147214 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:00.147729 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:00.147735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:00.148081 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:00.148532 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.148545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.149425 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.149435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.150298 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.150307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.150801 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.150805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.150975 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.151475 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:00.151491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:00.152933 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:00.154798 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.154804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.154817 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.155073 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.155414 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.155416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.155417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.155549 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.156035 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:00.156043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:00.156380 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 00:42:00.156610 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 00:42:00.156988 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260927 00:42:00.159202 8789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260927 00:42:00.159453 8789 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:00.159941 8789 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260927 00:42:00.161391 8598 ts_manager.cc:194] Registered new tserver with Master: f2239511d4f247a3a4265c6d524c242e (127.0.98.1:39607)
I20260927 00:42:00.162461 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33875
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:42:00.180069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:00.286788 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:00.287034 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:00.287081 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:00.291009 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:00.291066 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:00.291122 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:00.296234 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:00.297190 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:00.298211 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:00.307606 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.307693 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.307606 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.307807 8794 server_base.cc:1034] running on GCE node
I20260927 00:42:00.308365 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:00.309026 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:00.310199 8794 hybrid_clock.cc:648] HybridClock initialized: now 1790469720310182 us; error 28 us; skew 500 ppm
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:00.313084 8794 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:42:00.313963 8794 webserver.cc:460] Webserver started at http://127.0.98.2:39401/ using document root <none> and password file <none>
I20260927 00:42:00.314517 8794 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:00.314584 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:00.314746 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:00.316849 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "26bfcca19bef4379b30cbd4290593de7"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "a6fa480cd21dcf2b3575e9fdffd0657e"
server_key_iv: "dcb309e7eef8a88750553d5780cfe5d1"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.317448 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26bfcca19bef4379b30cbd4290593de7"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "a6fa480cd21dcf2b3575e9fdffd0657e"
server_key_iv: "dcb309e7eef8a88750553d5780cfe5d1"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.320976 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:42:00.323228 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.324397 8794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:00.324520 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "26bfcca19bef4379b30cbd4290593de7"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "a6fa480cd21dcf2b3575e9fdffd0657e"
server_key_iv: "dcb309e7eef8a88750553d5780cfe5d1"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.324615 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:00.339787 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:00.342938 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:00.343148 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:00.343737 8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:00.344620 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:00.344676 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.344743 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:00.344796 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.353323 8794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43333
I20260927 00:42:00.353433 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43333 every 8 connection(s)
I20260927 00:42:00.354192 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:00.355613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8794
I20260927 00:42:00.355764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:00.356120 392 external_mini_cluster.cc:1428] Setting key 8cd06226f837e5011f5fc3d7d5fa4f54
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:42:00.370239 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.356169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37275 (local address 127.0.98.62:42953)
0927 00:42:00.356333 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:00.356339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:00.357312 (+ 973us) server_negotiation.cc:407] Connection header received
0927 00:42:00.359507 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:00.359513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:00.359577 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:00.359668 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:00.360836 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.361732 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.363343 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.363557 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.363641 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.366510 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:00.366528 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:00.366530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:00.366551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:00.368019 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.368539 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.368544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.368546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.368624 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.368988 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.369049 (+ 61us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.369050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.369254 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:00.369365 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:00.369969 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 00:42:00.370120 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:42:00.371214 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.356570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:37275)
0927 00:42:00.357137 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:00.357197 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:00.359249 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:00.359912 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:00.359919 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:00.360252 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:00.360695 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.360708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.361912 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.361916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.363194 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.363212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.363735 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.363744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.364007 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.364645 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:00.364668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:00.366351 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:00.368147 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.368152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.368165 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.368438 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.368715 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.368719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.368721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.368849 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.369364 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:00.369370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:00.369661 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 00:42:00.369855 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 00:42:00.370139 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":121,"threads_started":1}
I20260927 00:42:00.372385 8923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260927 00:42:00.372660 8923 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:00.373193 8923 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260927 00:42:00.374183 8598 ts_manager.cc:194] Registered new tserver with Master: 26bfcca19bef4379b30cbd4290593de7 (127.0.98.2:43333)
I20260927 00:42:00.375118 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:42:00.387501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:00.497107 8928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:00.497359 8928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:00.497440 8928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:00.501667 8928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:00.501735 8928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:00.501852 8928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:00.506770 8928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:00.507721 8928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:00.508664 8928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:00.518368 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.518417 8936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.518417 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.518591 8928 server_base.cc:1034] running on GCE node
I20260927 00:42:00.519131 8928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:00.519733 8928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:00.520978 8928 hybrid_clock.cc:648] HybridClock initialized: now 1790469720520929 us; error 48 us; skew 500 ppm
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:00.523638 8928 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:42:00.524446 8928 webserver.cc:460] Webserver started at http://127.0.98.3:37237/ using document root <none> and password file <none>
I20260927 00:42:00.525029 8928 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:00.525120 8928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:00.525334 8928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:00.527303 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d4c77ba3fce4778b48fa56fbf3dfa84"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "1717cdb51f599c3b3a8396eef648cf64"
server_key_iv: "97c05ab2a09fd13e6121ae098de9dfe7"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.527846 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4c77ba3fce4778b48fa56fbf3dfa84"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "1717cdb51f599c3b3a8396eef648cf64"
server_key_iv: "97c05ab2a09fd13e6121ae098de9dfe7"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.531405 8928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:42:00.533624 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.534765 8928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:00.534909 8928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "6d4c77ba3fce4778b48fa56fbf3dfa84"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "1717cdb51f599c3b3a8396eef648cf64"
server_key_iv: "97c05ab2a09fd13e6121ae098de9dfe7"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.535015 8928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:00.555079 8928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:00.557737 8928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:00.557928 8928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:00.558470 8928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:00.559224 8928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:00.559293 8928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.559368 8928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:00.559417 8928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.568037 8928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35229
I20260927 00:42:00.568073 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35229 every 8 connection(s)
I20260927 00:42:00.568872 8928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:00.573117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 8928
I20260927 00:42:00.573242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:00.573519 392 external_mini_cluster.cc:1428] Setting key 3d3de79f3573b61110a9bcc4dc62e54e
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:00.582229 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.570711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56307 (local address 127.0.98.62:42953)
0927 00:42:00.570842 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:00.570845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:00.571610 (+ 765us) server_negotiation.cc:407] Connection header received
0927 00:42:00.573013 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:00.573016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:00.573060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:00.573130 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:00.574594 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.575386 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.576546 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.576753 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.576828 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.579194 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:00.579197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:00.579199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:00.579222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:00.580467 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.580955 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.580959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.580960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.581034 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:00.581315 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:00.581318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:00.581320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:00.581478 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:00.581553 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:00.582014 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 00:42:00.582119 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:42:00.582939 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:00.571067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:56307)
0927 00:42:00.571466 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:00.571506 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:00.572757 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:00.573339 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:00.573348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:00.573804 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:00.574456 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.574476 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.575509 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.575512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.576442 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:00.576451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:00.576870 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:00.576872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:00.577087 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:00.577664 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:00.577680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:00.579038 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:00.580568 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.580572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.580582 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.580816 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.581113 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:00.581115 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:00.581116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:00.581237 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:00.581561 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:00.581567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:00.581763 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 00:42:00.581933 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 00:42:00.582164 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260927 00:42:00.584013 9057 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42953
I20260927 00:42:00.584234 9057 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:00.584676 9057 heartbeater.cc:511] Master 127.0.98.62:42953 requested a full tablet report, sending...
I20260927 00:42:00.585726 8598 ts_manager.cc:194] Registered new tserver with Master: 6d4c77ba3fce4778b48fa56fbf3dfa84 (127.0.98.3:35229)
I20260927 00:42:00.586436 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56307
I20260927 00:42:00.587325 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 00:42:00.657857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8660
I20260927 00:42:00.663709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8794
I20260927 00:42:00.668924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8928
I20260927 00:42:00.673929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 8566
I20260927 00:42:00.680141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--webserver_interface=127.0.98.62
--webserver_port=34447
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:00.786530 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:00.786785 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:00.786825 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:00.790702 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:00.790750 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:00.790768 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:00.790784 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:00.790799 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:00.795806 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:00.796720 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:00.797729 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:00.805879 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.805936 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.805948 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.806053 9066 server_base.cc:1034] running on GCE node
I20260927 00:42:00.806798 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:00.807806 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:00.808960 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790469720808943 us; error 26 us; skew 500 ppm
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:00.811677 9066 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:42:00.812570 9066 webserver.cc:460] Webserver started at http://127.0.98.62:34447/ using document root <none> and password file <none>
I20260927 00:42:00.813164 9066 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:00.813237 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:00.817142 9066 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:42:00.819265 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:00.820552 9066 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:00.820701 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "c8811f80df7847d58b76cb0da928f3ce"
format_stamp: "Formatted at 2026-09-27 00:41:59 on dist-test-slave-pkws"
server_key: "af8e667663dd8f44024f1842735e518b"
server_key_iv: "6ca9f29b89e5d7c0fbf17cb8451371ed"
server_key_version: "encryptionkey@0"
I20260927 00:42:00.821290 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:00.832336 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:00.835063 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:00.835258 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:00.842237 9066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42953
I20260927 00:42:00.842252 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42953 every 8 connection(s)
I20260927 00:42:00.843221 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:00.845667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9066
I20260927 00:42:00.846172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39607
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=34587
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:42:00.847891 9134 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:42:00.849488 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Bootstrap starting.
I20260927 00:42:00.865860 9134 log.cc:824] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:00.872061 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:42:00.872414 9134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Bootstrap complete.
I20260927 00:42:00.876484 9134 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:42:00.876834 9134 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8811f80df7847d58b76cb0da928f3ce, State: Initialized, Role: FOLLOWER
I20260927 00:42:00.877378 9134 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:42:00.877491 9134 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:00.877570 9134 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:00.877667 9134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:42:00.879439 9134 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:42:00.879786 9134 leader_election.cc:302] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8811f80df7847d58b76cb0da928f3ce; no voters:
I20260927 00:42:00.880110 9134 leader_election.cc:288] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:42:00.880237 9138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:42:00.880468 9138 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [term 2 LEADER]: Becoming Leader. State: Replica: c8811f80df7847d58b76cb0da928f3ce, State: Running, Role: LEADER
I20260927 00:42:00.880759 9138 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } }
I20260927 00:42:00.881264 9134 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:00.881685 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8811f80df7847d58b76cb0da928f3ce. Latest consensus state: current_term: 2 leader_uuid: "c8811f80df7847d58b76cb0da928f3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260927 00:42:00.881801 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:00.882061 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8811f80df7847d58b76cb0da928f3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8811f80df7847d58b76cb0da928f3ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42953 } } }
I20260927 00:42:00.882169 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:00.885892 9154 catalog_manager.cc:1259] Loaded cluster ID: 99a2d0e721584968b0ada13566f49a9f
I20260927 00:42:00.885967 9154 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: loading cluster ID for follower catalog manager: success
I20260927 00:42:00.887579 9154 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: acquiring CA information for follower catalog manager: success
I20260927 00:42:00.888592 9154 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:42:00.888708 9142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:00.889386 9142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:00.889801 9142 catalog_manager.cc:1259] Loaded cluster ID: 99a2d0e721584968b0ada13566f49a9f
I20260927 00:42:00.889871 9142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:00.890435 9142 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:00.890818 9142 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8811f80df7847d58b76cb0da928f3ce: Loaded TSK: 0
I20260927 00:42:00.891366 9142 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:42:00.967708 9136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:00.967952 9136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:00.967991 9136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:00.971990 9136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:00.972038 9136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:00.972093 9136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:00.977123 9136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34587
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:00.978067 9136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:00.979023 9136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:00.989699 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.989873 9161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:00.990121 9163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:00.990293 9136 server_base.cc:1034] running on GCE node
I20260927 00:42:00.990729 9136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:00.991473 9136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:00.992666 9136 hybrid_clock.cc:648] HybridClock initialized: now 1790469720992648 us; error 31 us; skew 500 ppm
Sep 27 00:42:00 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469720, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:00.996011 9136 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 00:42:00.997006 9136 webserver.cc:460] Webserver started at http://127.0.98.1:34587/ using document root <none> and password file <none>
I20260927 00:42:00.997562 9136 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:00.997633 9136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:01.001535 9136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:01.003718 9170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.005064 9136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:01.005189 9136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f2239511d4f247a3a4265c6d524c242e"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "885e17c98892ad26a15d34ee6f1bbbb9"
server_key_iv: "625e537fb6a8e9fe514535f3544941ad"
server_key_version: "encryptionkey@0"
I20260927 00:42:01.005731 9136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:01.029294 9136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:01.032526 9136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:01.032714 9136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:01.033339 9136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:01.034241 9136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:01.034314 9136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.034394 9136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:01.034439 9136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.043149 9136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39607
I20260927 00:42:01.043231 9283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39607 every 8 connection(s)
I20260927 00:42:01.044129 9136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:01.052702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9136
I20260927 00:42:01.053234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43333
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=39401
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:42:01.057258 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.046192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50331 (local address 127.0.98.62:42953)
0927 00:42:01.046604 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.046625 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.046845 (+ 220us) server_negotiation.cc:407] Connection header received
0927 00:42:01.048340 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.048351 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.048631 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.048957 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.049969 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.050895 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.051984 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.052515 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.052630 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.056491 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50331: BlockingRecv error: recv got EOF from 127.0.98.1:50331 (error 108)
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":131,"threads_started":1}
I20260927 00:42:01.057266 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.046241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:50331)
0927 00:42:01.046686 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.046736 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.047982 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.049036 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.049042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.049378 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.049805 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.049818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.051009 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.051011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.051881 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.051890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.052670 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.052675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.052907 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.053575 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.053601 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.056288 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":136,"threads_started":1}
W20260927 00:42:01.057897 9284 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42953: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.063337 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.058380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:53633)
0927 00:42:01.058504 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.058520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.058597 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.058849 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.058852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.059007 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.059091 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.059096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.059810 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.059814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.060416 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.060425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.060919 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.060923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.061020 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.061497 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.061509 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.063201 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 00:42:01.063509 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.058474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53633 (local address 127.0.98.62:42953)
0927 00:42:01.058587 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.058591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.058605 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:01.058686 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.058689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.058760 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.058837 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.059204 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.059696 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.060537 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.060774 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.060942 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.063309 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53633: BlockingRecv error: recv got EOF from 127.0.98.1:53633 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.068758 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.063970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:45947)
0927 00:42:01.064055 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.064070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.064136 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.064343 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.064345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.064478 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.064540 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.064544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.065531 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.065534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.066030 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.066038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.066575 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.066578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.066633 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.067121 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.067132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.068679 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 00:42:01.068946 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.064036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45947 (local address 127.0.98.62:42953)
0927 00:42:01.064147 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.064151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.064166 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:42:01.064201 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.064203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.064240 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.064324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.064640 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.065432 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.066135 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.066469 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.066563 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.068810 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45947: BlockingRecv error: recv got EOF from 127.0.98.1:45947 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 00:42:01.069118 9284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:42:01.163072 9288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:01.163309 9288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:01.163349 9288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:01.167364 9288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:01.167413 9288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:01.167542 9288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:01.172631 9288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39401
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:01.173589 9288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:01.174562 9288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:01.183979 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:01.183949 9293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:01.183966 9294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:01.184085 9288 server_base.cc:1034] running on GCE node
I20260927 00:42:01.184650 9288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:01.185429 9288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:01.186615 9288 hybrid_clock.cc:648] HybridClock initialized: now 1790469721186587 us; error 46 us; skew 500 ppm
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:01.189429 9288 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 00:42:01.190171 9288 webserver.cc:460] Webserver started at http://127.0.98.2:39401/ using document root <none> and password file <none>
I20260927 00:42:01.190690 9288 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:01.190759 9288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:01.194341 9288 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:42:01.196250 9303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.197517 9288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:01.197605 9288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "26bfcca19bef4379b30cbd4290593de7"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "a6fa480cd21dcf2b3575e9fdffd0657e"
server_key_iv: "dcb309e7eef8a88750553d5780cfe5d1"
server_key_version: "encryptionkey@0"
I20260927 00:42:01.198076 9288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:01.216420 9288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:01.219512 9288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:01.219656 9288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:01.220160 9288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:01.220920 9288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:01.220965 9288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.221002 9288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:01.221024 9288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.229550 9288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43333
I20260927 00:42:01.229571 9416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43333 every 8 connection(s)
I20260927 00:42:01.230494 9288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.239764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9288
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.240262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35229
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42953
--webserver_port=37237
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:01.241530 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.232207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37403 (local address 127.0.98.62:42953)
0927 00:42:01.232346 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.232349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.233177 (+ 828us) server_negotiation.cc:407] Connection header received
0927 00:42:01.234494 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.234498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.234559 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.234667 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.235804 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.236328 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.237536 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.237760 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.237873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.241430 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37403: BlockingRecv error: recv got EOF from 127.0.98.2:37403 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:42:01.242233 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.232577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:37403)
0927 00:42:01.233030 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.233077 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.234281 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.234851 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.234858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.235219 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.235661 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.235674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.236453 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.236461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.237411 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.237420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.237905 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.237909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.238078 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.238585 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.238601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.241299 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
W20260927 00:42:01.242705 9417 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42953: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.248162 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.243100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:47143)
0927 00:42:01.243362 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.243373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.243494 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.243756 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.243759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.243911 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.243992 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.243996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.244694 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.244701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.245295 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.245303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.245762 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.245765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.245855 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.246327 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.246339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.248028 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 00:42:01.248256 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.243213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47143 (local address 127.0.98.62:42953)
0927 00:42:01.243330 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.243340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.243453 (+ 113us) server_negotiation.cc:407] Connection header received
0927 00:42:01.243605 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.243610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.243651 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.243737 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.244103 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.244587 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.245454 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.245653 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.245763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.248132 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47143: BlockingRecv error: recv got EOF from 127.0.98.2:47143 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.253237 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.248646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:44493)
0927 00:42:01.248730 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.248745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.248811 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.249082 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.249085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.249232 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.249293 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.249298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.249976 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.249982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.250489 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.250497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.250933 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.250936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.251014 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.251471 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.251481 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.253121 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 00:42:01.253444 9417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:42:01.253517 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.248733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44493 (local address 127.0.98.62:42953)
0927 00:42:01.248839 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.248845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.248871 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:42:01.248934 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.248939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.248984 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.249069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.249408 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.249872 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.250604 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.250822 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.250930 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.253240 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44493: BlockingRecv error: recv got EOF from 127.0.98.2:44493 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:42:01.350039 9420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:01.350286 9420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:01.350327 9420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:01.354285 9420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:01.354332 9420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:01.354421 9420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:01.359486 9420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37237
--tserver_master_addrs=127.0.98.62:42953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:01.360452 9420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:01.361425 9420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:01.370536 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:01.370502 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:01.370643 9420 server_base.cc:1034] running on GCE node
W20260927 00:42:01.370513 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:01.371090 9420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:01.371690 9420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:01.372848 9420 hybrid_clock.cc:648] HybridClock initialized: now 1790469721372838 us; error 30 us; skew 500 ppm
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:01.375569 9420 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 00:42:01.376475 9420 webserver.cc:460] Webserver started at http://127.0.98.3:37237/ using document root <none> and password file <none>
I20260927 00:42:01.377060 9420 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:01.377162 9420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:01.380934 9420 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:42:01.383427 9435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.384624 9420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:01.384764 9420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "6d4c77ba3fce4778b48fa56fbf3dfa84"
format_stamp: "Formatted at 2026-09-27 00:42:00 on dist-test-slave-pkws"
server_key: "1717cdb51f599c3b3a8396eef648cf64"
server_key_iv: "97c05ab2a09fd13e6121ae098de9dfe7"
server_key_version: "encryptionkey@0"
I20260927 00:42:01.385356 9420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:01.406414 9420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:01.409874 9420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:01.410138 9420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:01.410753 9420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:01.411597 9420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:01.411665 9420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.411736 9420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:01.411788 9420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:01.420991 9420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35229
I20260927 00:42:01.421039 9548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35229 every 8 connection(s)
I20260927 00:42:01.421999 9420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:01.427290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9420
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:42:01.433629 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.423811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43249 (local address 127.0.98.62:42953)
0927 00:42:01.423947 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.423951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.424852 (+ 901us) server_negotiation.cc:407] Connection header received
0927 00:42:01.426327 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.426331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.426388 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.426490 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.427906 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.428606 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.429804 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.430039 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.430151 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.433512 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43249: BlockingRecv error: recv got EOF from 127.0.98.3:43249 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:42:01.434247 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.424245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:43249)
0927 00:42:01.424677 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.424749 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.426110 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.426671 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.426678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.427084 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.427760 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.427776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.428728 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.428731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.429681 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.429690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.430188 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.430192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.430377 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.430974 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.430989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.433420 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
W20260927 00:42:01.434670 9549 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42953 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42953: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 00:42:01.437387 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.428369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.0.98.62:42953)
0927 00:42:01.428593 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.428596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.428608 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:42:01.428639 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.428641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.428680 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.428787 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.429553 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.430067 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.431085 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.431312 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.431425 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.433018 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:01.433023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:01.433037 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:01.433062 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:01.435166 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:01.435628 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:01.435636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:01.435645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:01.435792 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:01.436171 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:01.436173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:01.436174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:01.436496 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:01.436672 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:01.437008 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:42:01.437223 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.440442 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.435083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:58615)
0927 00:42:01.435251 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.435266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.435342 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.435607 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.435610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.435764 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.435883 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.435889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.436662 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.436665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.437511 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.437520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.437991 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.437994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.438080 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.438511 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.438522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.440351 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 00:42:01.440603 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.435160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58615 (local address 127.0.98.62:42953)
0927 00:42:01.435268 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.435272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.435305 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:42:01.435424 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.435427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.435472 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.435564 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.435982 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.436522 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.437623 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.437883 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.437962 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.440459 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58615: BlockingRecv error: recv got EOF from 127.0.98.3:58615 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:01 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:01.445473 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:01.440976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:42283)
0927 00:42:01.441087 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:01.441103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:01.441178 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:01.441392 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:01.441394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:01.441543 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:01.441613 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.441617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.442314 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.442320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.442836 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:01.442844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.443286 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.443289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:01.443363 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.443821 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:01.443835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:01.445388 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 00:42:01.445597 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:01.441002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42283 (local address 127.0.98.62:42953)
0927 00:42:01.441152 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:01.441155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:01.441167 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:42:01.441254 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:01.441257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:01.441290 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:01.441379 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:01.441720 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.442210 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.442952 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:01.443182 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:01.443285 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:01.445495 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42283: BlockingRecv error: recv got EOF from 127.0.98.3:42283 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 00:42:01.445763 9549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:02.075371 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:02.069747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:34925)
0927 00:42:02.070061 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:02.070075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:02.070178 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:02.070433 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:02.070436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:02.070619 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:02.070713 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.070718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.071455 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.071458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.072048 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.072057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.072493 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.072496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.072615 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.073262 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:02.073282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:02.075251 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
W20260927 00:42:02.075556 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:02.069820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34925 (local address 127.0.98.62:42953)
0927 00:42:02.070092 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:02.070095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:02.070138 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:42:02.070265 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:02.070267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:02.070307 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:02.070384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:02.070815 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.071339 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.072155 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.072364 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.072438 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.075424 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34925: BlockingRecv error: recv got EOF from 127.0.98.1:34925 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":64,"threads_started":1}
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:02.259505 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:02.253974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:39733)
0927 00:42:02.254274 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:02.254289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:02.254428 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:02.254712 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:02.254715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:02.254955 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:02.255065 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.255071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.255831 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.255836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.256414 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.256422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.256880 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.256884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.257011 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.257542 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:02.257553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:02.259389 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
W20260927 00:42:02.259644 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:02.254079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39733 (local address 127.0.98.62:42953)
0927 00:42:02.254245 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:02.254249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:02.254372 (+ 123us) server_negotiation.cc:407] Connection header received
0927 00:42:02.254544 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:02.254547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:02.254597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:02.254691 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:02.255186 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.255717 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.256540 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.256768 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.256878 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.259518 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39733: BlockingRecv error: recv got EOF from 127.0.98.2:39733 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:02 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:02.451519 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:02.446315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:45481)
0927 00:42:02.446586 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:02.446601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:02.446698 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:02.446954 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:02.446956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:02.447134 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:02.447245 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.447250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.447965 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.447968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.448507 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:02.448513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.449024 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.449027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:02.449103 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.449576 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:02.449587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:02.451399 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
W20260927 00:42:02.451706 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:02.446415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45481 (local address 127.0.98.62:42953)
0927 00:42:02.446597 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:02.446600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:02.446654 (+ 54us) server_negotiation.cc:407] Connection header received
0927 00:42:02.446774 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:02.446776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:02.446821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:02.446921 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:02.447342 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.447841 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.448616 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:02.448829 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:02.449022 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:02.451562 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45481: BlockingRecv error: recv got EOF from 127.0.98.3:45481 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:03.081189 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:03.076184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:43113)
0927 00:42:03.076453 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:03.076467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:03.076551 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:03.076793 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:03.076795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:03.077030 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:03.077125 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.077131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.077839 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.077841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.078360 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.078367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.078767 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.078769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.078826 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.079281 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:03.079291 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:03.081053 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":111,"threads_started":1}
W20260927 00:42:03.081343 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:03.076254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43113 (local address 127.0.98.62:42953)
0927 00:42:03.076519 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:03.076522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:03.076535 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:03.076630 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:03.076632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:03.076672 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:03.076740 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:03.077235 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.077749 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.078467 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.078673 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.078747 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.081176 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43113: BlockingRecv error: recv got EOF from 127.0.98.1:43113 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:03.265868 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:03.260285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:39217)
0927 00:42:03.260561 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:03.260574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:03.260677 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:03.260948 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:03.260950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:03.261135 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:03.261226 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.261231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.261957 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.261960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.262520 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.262527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.262926 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.262928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.262987 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.263621 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:03.263640 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:03.265735 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260927 00:42:03.265982 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:03.260329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39217 (local address 127.0.98.62:42953)
0927 00:42:03.260498 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:03.260503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:03.260662 (+ 159us) server_negotiation.cc:407] Connection header received
0927 00:42:03.260755 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:03.260758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:03.260801 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:03.260860 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:03.261327 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.261847 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.262620 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.262829 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.262906 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.265869 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39217: BlockingRecv error: recv got EOF from 127.0.98.2:39217 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:03 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:03.457545 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:03.452422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:39997)
0927 00:42:03.452747 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:03.452759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:03.452854 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:03.453125 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:03.453127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:03.453297 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:03.453411 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.453417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.454096 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.454099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.454661 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:03.454668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.455053 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.455055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:03.455115 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.455571 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:03.455582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:03.457354 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
W20260927 00:42:03.457645 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:03.452641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39997 (local address 127.0.98.62:42953)
0927 00:42:03.452758 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:03.452762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:03.452836 (+ 74us) server_negotiation.cc:407] Connection header received
0927 00:42:03.452971 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:03.452973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:03.453016 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:03.453092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:03.453506 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.454004 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.454758 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:03.454964 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:03.455036 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:03.457498 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39997: BlockingRecv error: recv got EOF from 127.0.98.3:39997 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:04.087670 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:04.082026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:50225)
0927 00:42:04.082270 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:04.082283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:04.082403 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:04.082682 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:04.082687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:04.082872 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:04.082990 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.082996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.083754 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.083756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.084341 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.084349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.084754 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.084756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.084814 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.085336 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:04.085348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:04.087479 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
W20260927 00:42:04.087736 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:04.082041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50225 (local address 127.0.98.62:42953)
0927 00:42:04.082291 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:04.082295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:04.082330 (+ 35us) server_negotiation.cc:407] Connection header received
0927 00:42:04.082496 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:04.082498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:04.082539 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:04.082627 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:04.083132 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.083658 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.084443 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.084663 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.084737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.087597 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50225: BlockingRecv error: recv got EOF from 127.0.98.1:50225 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":114,"threads_started":1}
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:04.272284 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:04.266744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:34245)
0927 00:42:04.267028 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:04.267041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:04.267140 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:04.267369 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:04.267371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:04.267545 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:04.267649 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.267654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.268531 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.268535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.269206 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.269215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.269698 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.269702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.269789 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.270335 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:04.270347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:04.272158 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
W20260927 00:42:04.272436 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:04.266802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34245 (local address 127.0.98.62:42953)
0927 00:42:04.266945 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:04.266949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:04.267115 (+ 166us) server_negotiation.cc:407] Connection header received
0927 00:42:04.267220 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:04.267223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:04.267267 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:04.267337 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:04.267741 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.268352 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.269330 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.269570 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.269698 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.272285 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34245: BlockingRecv error: recv got EOF from 127.0.98.2:34245 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:04 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:04.463825 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:04.458481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:60001)
0927 00:42:04.458768 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:04.458781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:04.458879 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:04.459108 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:04.459110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:04.459282 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:04.459372 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.459377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.460111 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.460113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.460683 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:04.460691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.461100 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.461103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:04.461162 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.461620 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:04.461630 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:04.463648 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":116,"threads_started":1}
W20260927 00:42:04.463884 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:04.458530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60001 (local address 127.0.98.62:42953)
0927 00:42:04.458684 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:04.458688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:04.458853 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:42:04.458956 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:04.458958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:04.459003 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:04.459071 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:04.459506 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.460017 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.460781 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:04.461003 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:04.461077 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:04.463776 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60001: BlockingRecv error: recv got EOF from 127.0.98.3:60001 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:05.099042 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:05.088521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:38813)
0927 00:42:05.088800 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:05.088814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:05.088951 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:05.089317 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:05.089320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:05.089511 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:05.089622 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.089628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.090353 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.090356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.090917 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.090924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.091373 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.091377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.091471 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.096974 (+ 5503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:05.096987 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:05.098870 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":67,"threads_started":1}
W20260927 00:42:05.099113 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:05.088572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38813 (local address 127.0.98.62:42953)
0927 00:42:05.088848 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:05.088851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:05.088865 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:05.089115 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:05.089118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:05.089161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:05.089292 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:05.089731 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.090261 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.091025 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.091243 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.091368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.098989 (+ 7621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38813: BlockingRecv error: recv got EOF from 127.0.98.1:38813 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 00:42:05.278960 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:05.273240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60717 (local address 127.0.98.62:42953)
0927 00:42:05.273383 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:05.273387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:05.273596 (+ 209us) server_negotiation.cc:407] Connection header received
0927 00:42:05.273728 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:05.273731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:05.273779 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:05.273883 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:05.274393 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.274972 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.275846 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.276097 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.276221 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.278857 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60717: BlockingRecv error: recv got EOF from 127.0.98.2:60717 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:42:05.278975 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:05.273187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:60717)
0927 00:42:05.273492 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:05.273505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:05.273628 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:05.273952 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:05.273956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:05.274165 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:05.274263 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.274269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.275101 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.275105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.275718 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.275726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.276220 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.276224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.276320 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.276815 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:05.276828 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:05.278734 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:05 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:05.470094 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:05.464641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:52553)
0927 00:42:05.464966 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:05.464978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:05.465103 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:05.465394 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:05.465396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:05.465598 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:05.465688 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.465693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.466428 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.466433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.467049 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:05.467061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.467496 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.467500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:05.467587 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.468132 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:05.468156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:05.469959 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
W20260927 00:42:05.470328 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:05.464708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52553 (local address 127.0.98.62:42953)
0927 00:42:05.464877 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:05.464882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:05.465051 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:42:05.465220 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:05.465223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:05.465278 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:05.465368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:05.465805 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.466316 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.467150 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:05.467381 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:05.467493 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:05.470099 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52553: BlockingRecv error: recv got EOF from 127.0.98.3:52553 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:06.105507 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:06.099844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:58271)
0927 00:42:06.100100 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:06.100123 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:06.100209 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:06.100448 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:06.100450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:06.100642 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:06.100742 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.100747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.101589 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.101591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.102320 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.102335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.102854 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.102860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.102988 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.103529 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:06.103540 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:06.105351 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":131,"threads_started":1}
W20260927 00:42:06.105581 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:06.100035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58271 (local address 127.0.98.62:42953)
0927 00:42:06.100248 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:06.100251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:06.100263 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:42:06.100295 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:06.100297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:06.100336 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:06.100416 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:06.100864 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.101489 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.102441 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.102723 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.102838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.105481 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58271: BlockingRecv error: recv got EOF from 127.0.98.1:58271 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:06.285719 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:06.279789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:56789)
0927 00:42:06.280089 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:06.280106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:06.280233 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:06.280503 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:06.280506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:06.280714 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:06.280805 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.280810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.281583 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.281589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.282337 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.282346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.282792 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.282795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.282880 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.283471 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:06.283485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:06.285593 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
W20260927 00:42:06.285874 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:06.279952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56789 (local address 127.0.98.62:42953)
0927 00:42:06.280122 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:06.280129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:06.280173 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:42:06.280336 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:06.280340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:06.280394 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:06.280481 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:06.280944 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.281471 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.282457 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.282682 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.282789 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.285725 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56789: BlockingRecv error: recv got EOF from 127.0.98.2:56789 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:06 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:06.476123 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:06.471017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:55783)
0927 00:42:06.471269 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:06.471282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:06.471375 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:06.471599 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:06.471601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:06.471784 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:06.471874 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.471879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.472606 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.472608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.473203 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:06.473211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.473626 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.473628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:06.473698 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.474168 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:06.474179 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:06.476006 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
W20260927 00:42:06.476362 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:06.471104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55783 (local address 127.0.98.62:42953)
0927 00:42:06.471275 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:06.471280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:06.471356 (+ 76us) server_negotiation.cc:407] Connection header received
0927 00:42:06.471454 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:06.471456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:06.471498 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:06.471565 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:06.471981 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.472511 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.473319 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:06.473529 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:06.473611 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:06.476227 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55783: BlockingRecv error: recv got EOF from 127.0.98.3:55783 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:07.111733 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:07.106335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:58917)
0927 00:42:07.106602 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:07.106623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:07.106745 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:07.107115 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:07.107117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:07.107300 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:07.107393 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.107398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.108160 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.108170 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.108710 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.108717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.109220 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.109223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.109287 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.109859 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:07.109870 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:07.111618 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
W20260927 00:42:07.111877 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:07.106380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58917 (local address 127.0.98.62:42953)
0927 00:42:07.106625 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:07.106630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:07.106716 (+ 86us) server_negotiation.cc:407] Connection header received
0927 00:42:07.106909 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:07.106912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:07.106997 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:07.107087 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:07.107501 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.108068 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.108817 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.109114 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.109200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.111750 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58917: BlockingRecv error: recv got EOF from 127.0.98.1:58917 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":60,"threads_started":1}
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:07.291838 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:07.286463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:46623)
0927 00:42:07.286834 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:07.286849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:07.286956 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:07.287181 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:07.287183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:07.287350 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:07.287442 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.287447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.288221 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.288224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.288855 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.288864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.289324 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.289326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.289388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.289885 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:07.289896 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:07.291729 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":79,"threads_started":1}
W20260927 00:42:07.291977 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:07.286504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46623 (local address 127.0.98.62:42953)
0927 00:42:07.286637 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:07.286641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:07.286972 (+ 331us) server_negotiation.cc:407] Connection header received
0927 00:42:07.287029 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:07.287031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:07.287077 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:07.287139 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:07.287547 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.288097 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.289001 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.289221 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.289337 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.291860 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46623: BlockingRecv error: recv got EOF from 127.0.98.2:46623 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:07 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:07.483057 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:07.476836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:44573)
0927 00:42:07.477146 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:07.477160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:07.477301 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:07.477575 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:07.477577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:07.477800 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:07.477890 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.477896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.478674 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.478679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.479239 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:07.479247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.479933 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.479937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:07.480022 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.480691 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:07.480707 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:07.482892 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":123,"threads_started":1}
W20260927 00:42:07.483126 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:07.476938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44573 (local address 127.0.98.62:42953)
0927 00:42:07.477107 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:07.477112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:07.477250 (+ 138us) server_negotiation.cc:407] Connection header received
0927 00:42:07.477411 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:07.477414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:07.477469 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:07.477555 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:07.478017 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.478556 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.479362 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:07.479782 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:07.479933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:07.483009 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44573: BlockingRecv error: recv got EOF from 127.0.98.3:44573 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:08.117871 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:08.112560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:44197)
0927 00:42:08.112864 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:08.112879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:08.112984 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:08.113235 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:08.113237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:08.113411 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:08.113517 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.113522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.114231 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.114233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.114797 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.114804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.115322 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.115325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.115388 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.115886 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:08.115897 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:08.117762 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
W20260927 00:42:08.118074 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:08.112646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44197 (local address 127.0.98.62:42953)
0927 00:42:08.112962 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:08.112966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:08.112979 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:08.113070 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:08.113072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:08.113116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:08.113193 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:08.113617 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.114133 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.114978 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.115213 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.115303 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.117893 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44197: BlockingRecv error: recv got EOF from 127.0.98.1:44197 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:08.297974 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:08.292645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:35807)
0927 00:42:08.293090 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:08.293110 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:08.293205 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:08.293477 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:08.293479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:08.293673 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:08.293770 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.293776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.294490 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.294492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.295081 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.295089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.295492 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.295495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.295555 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.296046 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:08.296058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:08.297862 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":76,"threads_started":1}
W20260927 00:42:08.298110 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:08.292741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35807 (local address 127.0.98.62:42953)
0927 00:42:08.292941 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:08.292946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:08.293186 (+ 240us) server_negotiation.cc:407] Connection header received
0927 00:42:08.293288 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:08.293290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:08.293340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:08.293401 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:08.293878 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.294394 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.295188 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.295397 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.295473 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.297991 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35807: BlockingRecv error: recv got EOF from 127.0.98.2:35807 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:08 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:08.489498 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:08.483717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:33295)
0927 00:42:08.484001 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:08.484017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:08.484161 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:08.484446 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:08.484448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:08.484631 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:08.484724 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.484729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.485509 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.485513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.486143 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:08.486152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.486585 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.486588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:08.486654 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.487124 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:08.487135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:08.489328 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
W20260927 00:42:08.489547 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:08.483810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33295 (local address 127.0.98.62:42953)
0927 00:42:08.483970 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:08.483974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:08.484117 (+ 143us) server_negotiation.cc:407] Connection header received
0927 00:42:08.484266 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:08.484268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:08.484318 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:08.484385 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:08.484822 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.485380 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.486262 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:08.486482 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:08.486560 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:08.489446 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33295: BlockingRecv error: recv got EOF from 127.0.98.3:33295 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:09.123857 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:09.118583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:39513)
0927 00:42:09.118876 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:09.118891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:09.118978 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:09.119228 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:09.119230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:09.119404 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:09.119523 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.119529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.120258 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.120260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.120830 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.120839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.121310 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.121313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.121375 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.121845 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:09.121856 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:09.123680 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
W20260927 00:42:09.123914 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:09.118752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39513 (local address 127.0.98.62:42953)
0927 00:42:09.118952 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:09.118955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:09.118968 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:09.119063 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:09.119065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:09.119116 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:09.119193 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:09.119625 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.120155 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.120990 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.121214 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.121291 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.123798 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39513: BlockingRecv error: recv got EOF from 127.0.98.1:39513 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:09.304461 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:09.298728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:48945)
0927 00:42:09.299017 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:09.299032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:09.299181 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:09.299484 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:09.299487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:09.299712 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:09.299803 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.299809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.300803 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.300808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.301520 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.301529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.301990 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.301994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.302083 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.302575 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:09.302587 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:09.304335 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
W20260927 00:42:09.304628 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:09.298795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48945 (local address 127.0.98.62:42953)
0927 00:42:09.298978 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:09.298983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:09.299120 (+ 137us) server_negotiation.cc:407] Connection header received
0927 00:42:09.299317 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:09.299320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:09.299373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:09.299462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:09.299920 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.300693 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.301637 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.301863 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.301990 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.304473 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48945: BlockingRecv error: recv got EOF from 127.0.98.2:48945 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:09 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:09.495621 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:09.490373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:38751)
0927 00:42:09.490714 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:09.490729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:09.490851 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:09.491168 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:09.491174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:09.491428 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:09.491557 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.491563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.492266 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.492268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.492809 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:09.492816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.493263 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.493266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:09.493347 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.493798 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:09.493808 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:09.495510 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":85,"threads_started":1}
W20260927 00:42:09.495791 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:09.490425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38751 (local address 127.0.98.62:42953)
0927 00:42:09.490584 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:09.490590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:09.490822 (+ 232us) server_negotiation.cc:407] Connection header received
0927 00:42:09.490952 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:09.490955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:09.491019 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:09.491100 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:09.491655 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.492174 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.492957 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:09.493172 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:09.493248 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:09.495637 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38751: BlockingRecv error: recv got EOF from 127.0.98.3:38751 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:10.135143 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:10.124655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:51299)
0927 00:42:10.124972 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:10.124989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:10.125092 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:10.125333 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:10.125335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:10.125507 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:10.125603 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.125607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.126417 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.126420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.126979 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.126987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.127390 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.127393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.127473 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.132983 (+ 5510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:10.132996 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:10.135031 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
W20260927 00:42:10.135329 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:10.124729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51299 (local address 127.0.98.62:42953)
0927 00:42:10.125026 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:10.125030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:10.125043 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:10.125179 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:10.125181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:10.125228 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:10.125297 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:10.125709 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.126318 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.127083 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.127294 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.127370 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.135180 (+ 7810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51299: BlockingRecv error: recv got EOF from 127.0.98.1:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:10.310849 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:10.305255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:53989)
0927 00:42:10.305512 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:10.305525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:10.305620 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:10.305838 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:10.305841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:10.306019 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:10.306112 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.306117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.306804 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.306807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.307347 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.307354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.307785 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.307788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.307847 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.308467 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:10.308513 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:10.310673 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
W20260927 00:42:10.310925 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:10.305295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53989 (local address 127.0.98.62:42953)
0927 00:42:10.305435 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:10.305439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:10.305593 (+ 154us) server_negotiation.cc:407] Connection header received
0927 00:42:10.305684 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:10.305686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:10.305727 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:10.305802 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:10.306204 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.306713 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.307449 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.307686 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.307768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.310810 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53989: BlockingRecv error: recv got EOF from 127.0.98.2:53989 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:10 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:10.502138 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:10.496339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:55959)
0927 00:42:10.496594 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:10.496607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:10.496697 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:10.497077 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:10.497079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:10.497253 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:10.497363 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.497373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.498389 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.498395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.499038 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:10.499048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.499484 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.499486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:10.499544 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.500046 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:10.500057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:10.502027 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
W20260927 00:42:10.502404 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:10.496430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55959 (local address 127.0.98.62:42953)
0927 00:42:10.496646 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:10.496649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:10.496663 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:10.496835 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:10.496839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:10.496950 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:10.497059 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:10.497525 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.498268 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.499149 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:10.499386 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:10.499490 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:10.502210 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55959: BlockingRecv error: recv got EOF from 127.0.98.3:55959 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:11.141764 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:11.136164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:48951)
0927 00:42:11.136422 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:11.136436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:11.136525 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:11.136791 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:11.136793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:11.137037 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:11.137136 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.137142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.137865 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.137868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.138466 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.138476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.138877 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.138879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.138942 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.139512 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:11.139525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:11.141654 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
W20260927 00:42:11.141929 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:11.136263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48951 (local address 127.0.98.62:42953)
0927 00:42:11.136488 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:11.136491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:11.136506 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:42:11.136617 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:11.136619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:11.136661 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:11.136756 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:11.137238 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.137769 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.138567 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.138782 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.138853 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.141798 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48951: BlockingRecv error: recv got EOF from 127.0.98.1:48951 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:11.317142 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:11.311678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:37411)
0927 00:42:11.312051 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:11.312067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:11.312208 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:11.312516 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:11.312527 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:11.312740 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:11.312837 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.312842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.313596 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.313598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.314165 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.314172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.314581 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.314583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.314645 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.315257 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:11.315270 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:11.317029 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":97,"threads_started":1}
W20260927 00:42:11.317281 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:11.311723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37411 (local address 127.0.98.62:42953)
0927 00:42:11.311874 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:11.311879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:11.312162 (+ 283us) server_negotiation.cc:407] Connection header received
0927 00:42:11.312323 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:11.312326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:11.312381 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:11.312486 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:11.312961 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.313491 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.314266 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.314473 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.314547 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.317164 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37411: BlockingRecv error: recv got EOF from 127.0.98.2:37411 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:11 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:11.508675 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:11.502901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:32963)
0927 00:42:11.503236 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:11.503252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:11.503383 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:11.503650 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:11.503653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:11.503863 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:11.503949 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.503955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.504717 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.504720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.505668 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:11.505678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.506125 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.506128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:11.506192 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.506665 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:11.506675 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:11.508499 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":136,"threads_started":1}
W20260927 00:42:11.508785 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:11.503083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32963 (local address 127.0.98.62:42953)
0927 00:42:11.503230 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:11.503234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:11.503335 (+ 101us) server_negotiation.cc:407] Connection header received
0927 00:42:11.503488 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:11.503492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:11.503543 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:11.503629 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:11.504067 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.504585 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.505787 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:11.506000 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:11.506080 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:11.508625 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32963: BlockingRecv error: recv got EOF from 127.0.98.3:32963 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:12.148068 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:12.142583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:38777)
0927 00:42:12.142844 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:12.142857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:12.142942 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:12.143213 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:12.143216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:12.143482 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:12.143605 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.143612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.144396 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.144401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.145051 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.145063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.145551 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.145555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.145645 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.146117 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:12.146129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:12.147941 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
W20260927 00:42:12.148293 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:12.142694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38777 (local address 127.0.98.62:42953)
0927 00:42:12.142938 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:12.142941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:12.142955 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:12.143015 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:12.143017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:12.143059 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:12.143177 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:12.143737 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.144276 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.145178 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.145418 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.145530 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.148123 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38777: BlockingRecv error: recv got EOF from 127.0.98.1:38777 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:12.323076 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:12.317943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:60637)
0927 00:42:12.318206 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:12.318220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:12.318322 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:12.318573 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:12.318575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:12.318756 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:12.318849 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.318854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.319553 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.319556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.320109 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.320117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.320525 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.320543 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.320603 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.321117 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:12.321130 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:12.322930 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
W20260927 00:42:12.323150 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:12.318078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60637 (local address 127.0.98.62:42953)
0927 00:42:12.318235 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:12.318239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:12.318292 (+ 53us) server_negotiation.cc:407] Connection header received
0927 00:42:12.318405 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:12.318407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:12.318450 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:12.318516 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:12.318945 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.319449 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.320217 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.320432 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.320506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.323062 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60637: BlockingRecv error: recv got EOF from 127.0.98.2:60637 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:12 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:12.514870 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:12.509492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:38795)
0927 00:42:12.509846 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:12.509858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:12.509952 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:12.510212 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:12.510214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:12.510460 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:12.510560 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.510566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.511435 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.511438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.511980 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:12.511987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.512433 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.512435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:12.512496 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.512992 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:12.513003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:12.514755 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":108,"threads_started":1}
W20260927 00:42:12.515067 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:12.509569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38795 (local address 127.0.98.62:42953)
0927 00:42:12.509712 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:12.509718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:12.509946 (+ 228us) server_negotiation.cc:407] Connection header received
0927 00:42:12.510058 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:12.510060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:12.510107 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:12.510175 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:12.510705 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.511333 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.512093 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:12.512330 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:12.512425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:12.514902 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38795: BlockingRecv error: recv got EOF from 127.0.98.3:38795 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:13.154443 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:13.148881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:38455)
0927 00:42:13.149313 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:13.149327 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:13.149472 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:13.149687 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:13.149690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:13.149872 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:13.149965 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.149970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.150701 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.150703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.151252 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.151259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.151672 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.151674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.151758 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.152408 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:13.152420 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:13.154334 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":76,"threads_started":1}
W20260927 00:42:13.154639 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:13.148958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38455 (local address 127.0.98.62:42953)
0927 00:42:13.149210 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:13.149217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:13.149418 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:42:13.149539 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:13.149541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:13.149585 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:13.149658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:13.150089 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.150611 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.151347 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.151576 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.151652 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.154483 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38455: BlockingRecv error: recv got EOF from 127.0.98.1:38455 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:13.329568 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:13.323953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:40841)
0927 00:42:13.324286 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:13.324300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:13.324441 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:13.324721 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:13.324724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:13.325005 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:13.325117 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.325123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.325894 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.325899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.326462 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.326470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.326931 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.326935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.327022 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.327544 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:13.327556 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:13.329399 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
W20260927 00:42:13.329624 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:13.324012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40841 (local address 127.0.98.62:42953)
0927 00:42:13.324240 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:13.324245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:13.324387 (+ 142us) server_negotiation.cc:407] Connection header received
0927 00:42:13.324549 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:13.324553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:13.324605 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:13.324692 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:13.325243 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.325782 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.326584 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.326818 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.326931 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.329518 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40841: BlockingRecv error: recv got EOF from 127.0.98.2:40841 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:13 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:13.520612 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:13.515572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:45179)
0927 00:42:13.515829 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:13.515842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:13.515937 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:13.516192 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:13.516194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:13.516371 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:13.516461 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.516466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.517225 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.517228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.517770 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:13.517777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.518188 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.518190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:13.518248 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.518746 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:13.518757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:13.520501 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260927 00:42:13.520748 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:13.515627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45179 (local address 127.0.98.62:42953)
0927 00:42:13.515775 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:13.515782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:13.515913 (+ 131us) server_negotiation.cc:407] Connection header received
0927 00:42:13.516043 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:13.516045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:13.516091 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:13.516158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:13.516566 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.517127 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.517866 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:13.518094 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:13.518173 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:13.520632 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45179: BlockingRecv error: recv got EOF from 127.0.98.3:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:14.160429 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:14.155170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:60511)
0927 00:42:14.155432 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:14.155463 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:14.155553 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:14.155778 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:14.155780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:14.155953 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:14.156047 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.156052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.156781 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.156783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.157351 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.157358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.157861 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.157864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.157934 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.158429 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:14.158447 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:14.160318 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
W20260927 00:42:14.160634 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:14.155293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60511 (local address 127.0.98.62:42953)
0927 00:42:14.155529 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:14.155533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:14.155546 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:42:14.155631 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:14.155633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:14.155674 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:14.155741 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:14.156154 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.156691 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.157480 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.157725 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.157824 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.160453 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60511: BlockingRecv error: recv got EOF from 127.0.98.1:60511 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:14.335665 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:14.330302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:44565)
0927 00:42:14.330564 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:14.330577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:14.330672 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:14.330899 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:14.330902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:14.331084 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:14.331176 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.331181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.331919 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.331921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.332470 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.332477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.332872 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.332874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.332963 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.333524 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:14.333535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:14.335528 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
W20260927 00:42:14.335822 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:14.330440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44565 (local address 127.0.98.62:42953)
0927 00:42:14.330573 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:14.330576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:14.330625 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:42:14.330754 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:14.330756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:14.330797 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:14.330863 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:14.331295 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.331828 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.332566 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.332778 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.332853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.335674 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44565: BlockingRecv error: recv got EOF from 127.0.98.2:44565 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:14 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:14.527280 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:14.521393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:59371)
0927 00:42:14.521743 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:14.521757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:14.521891 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:14.522204 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:14.522207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:14.522404 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:14.522524 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.522530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.523285 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.523290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.523860 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:14.523868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.524335 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.524339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:14.524434 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.524975 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:14.524989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:14.527152 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
W20260927 00:42:14.527534 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:14.521435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59371 (local address 127.0.98.62:42953)
0927 00:42:14.521584 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:14.521588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:14.521836 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:42:14.522037 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:14.522040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:14.522094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:14.522179 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:14.522638 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.523174 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.523982 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:14.524222 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:14.524333 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:14.527369 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59371: BlockingRecv error: recv got EOF from 127.0.98.3:59371 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:15.171247 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:15.161213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:43119)
0927 00:42:15.161486 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:15.161505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:15.161625 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:15.161918 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:15.161921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:15.162113 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:15.162212 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.162218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.162951 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.162953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.163572 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.163580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.163989 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.163992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.164054 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.169152 (+ 5098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:15.169165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:15.171055 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
W20260927 00:42:15.171300 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:15.161343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43119 (local address 127.0.98.62:42953)
0927 00:42:15.161646 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:15.161649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:15.161661 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:42:15.161694 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:15.161696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:15.161736 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:15.161812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:15.162324 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.162845 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.163679 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.163890 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.163965 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.171175 (+ 7210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43119: BlockingRecv error: recv got EOF from 127.0.98.1:43119 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":75,"threads_started":1}
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:15.341825 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:15.336499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:55807)
0927 00:42:15.336796 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:15.336811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:15.336938 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:15.337222 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:15.337225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:15.337404 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:15.337494 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.337499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.338201 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.338203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.338746 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.338752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.339137 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.339140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.339205 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.339765 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:15.339777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:15.341693 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
W20260927 00:42:15.341957 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:15.336664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55807 (local address 127.0.98.62:42953)
0927 00:42:15.336830 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:15.336834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:15.336863 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:42:15.337030 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:15.337032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:15.337082 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:15.337151 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:15.337603 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.338110 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.338839 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.339041 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.339122 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.341825 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55807: BlockingRecv error: recv got EOF from 127.0.98.2:55807 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:15 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:15.534019 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:15.528317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.3:39565)
0927 00:42:15.528680 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:15.528693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:15.528824 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:15.529150 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:15.529152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:15.529356 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:15.529445 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.529450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.530197 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.530202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.530774 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:15.530782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.531242 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.531245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:15.531346 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.531849 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:15.531860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:15.533884 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
W20260927 00:42:15.534149 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:15.528368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39565 (local address 127.0.98.62:42953)
0927 00:42:15.528520 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:15.528526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:15.528774 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:42:15.528974 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:15.528978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:15.529033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:15.529127 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:15.529561 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.530086 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.530895 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:15.531128 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:15.531239 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:15.534010 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39565: BlockingRecv error: recv got EOF from 127.0.98.3:39565 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 00:42:16 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:16 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:16.177588 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:16.172229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.1:41621)
0927 00:42:16.172504 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:16.172527 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:16.172618 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:16.172853 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:16.172855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:16.173085 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:16.173178 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.173183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.173899 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.173901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.174434 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.174440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.174897 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.174900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.174958 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.175423 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:16.175434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:16.177423 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
W20260927 00:42:16.177659 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:16.172270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41621 (local address 127.0.98.62:42953)
0927 00:42:16.172533 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:16.172537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:16.172576 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:42:16.172702 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:16.172705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:16.172745 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:16.172819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:16.173278 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.173807 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.174556 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.174796 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.174884 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.177535 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41621: BlockingRecv error: recv got EOF from 127.0.98.1:41621 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
Sep 27 00:42:16 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 00:42:16 dist-test-slave-pkws krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 00:42:16.348120 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:16.342718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42953 (local address 127.0.98.2:49993)
0927 00:42:16.343026 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:16.343039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:16.343132 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:16.343344 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:16.343347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:16.343536 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:16.343629 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.343634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.344430 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.344433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.345068 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.345077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.345483 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.345486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.345554 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.346034 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:16.346045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:16.347938 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42953: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":134,"threads_started":1}
W20260927 00:42:16.348156 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:16.342779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49993 (local address 127.0.98.62:42953)
0927 00:42:16.342952 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:16.342957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:16.343117 (+ 160us) server_negotiation.cc:407] Connection header received
0927 00:42:16.343203 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:16.343205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:16.343245 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:16.343311 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:16.343726 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.344330 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.345170 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.345390 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.345470 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.348052 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49993: BlockingRecv error: recv got EOF from 127.0.98.2:49993 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 00:42:16.429503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9136
I20260927 00:42:16.435051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9288
I20260927 00:42:16.440141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9420
I20260927 00:42:16.445415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9066
2026-09-27T00:42:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17907 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T00:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:16Z Disabled control of system clock
I20260927 00:42:16.474890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 00:42:16.581849 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:16.582093 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:16.582126 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:16.585940 9616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:16.585987 9616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:16.586002 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:16.586019 9616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:16.586035 9616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:16.590413 9616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:16.591292 9616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:16.592535 9616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:16.601145 9624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:16.601116 9621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:16.601105 9622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:16.601163 9616 server_base.cc:1034] running on GCE node
I20260927 00:42:16.601889 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:16.603015 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:16.604197 9616 hybrid_clock.cc:648] HybridClock initialized: now 1790469736604184 us; error 27 us; skew 500 ppm
I20260927 00:42:16.605988 9616 webserver.cc:460] Webserver started at http://127.0.98.62:42207/ using document root <none> and password file <none>
I20260927 00:42:16.606606 9616 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:16.606698 9616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:16.606918 9616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:16.608949 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "ba52ad0d679841b49b55516affc2622b"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "b282a6beb9b007156f2e9f14f1bddfdb"
server_key_iv: "0841ae6459ed6f6f9e0c6e8b90d2c33b"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.609534 9616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba52ad0d679841b49b55516affc2622b"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "b282a6beb9b007156f2e9f14f1bddfdb"
server_key_iv: "0841ae6459ed6f6f9e0c6e8b90d2c33b"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.613039 9616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:42:16.615315 9630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:16.616583 9616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:16.616750 9616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "ba52ad0d679841b49b55516affc2622b"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "b282a6beb9b007156f2e9f14f1bddfdb"
server_key_iv: "0841ae6459ed6f6f9e0c6e8b90d2c33b"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.616853 9616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:16.642232 9616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:16.642901 9616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:16.643093 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:16.650007 9616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44775
I20260927 00:42:16.650017 9682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44775 every 8 connection(s)
I20260927 00:42:16.650928 9616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:16.653916 9683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:16.659200 9683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b: Bootstrap starting.
I20260927 00:42:16.660495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9616
I20260927 00:42:16.660610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:42:16.660908 392 external_mini_cluster.cc:1428] Setting key 98a88c94939a2d3f4504b53edb97f5f1
I20260927 00:42:16.662022 9683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:16.662796 9683 log.cc:824] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:16.664706 9683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b: No bootstrap required, opened a new log
I20260927 00:42:16.668223 9683 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260927 00:42:16.668418 9683 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:16.668475 9683 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba52ad0d679841b49b55516affc2622b, State: Initialized, Role: FOLLOWER
I20260927 00:42:16.669005 9683 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260927 00:42:16.669137 9683 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:16.669219 9683 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:16.669329 9683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:16.669843 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:16.661980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.0.98.62:44775)
0927 00:42:16.662600 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:16.662614 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:16.662647 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:42:16.663391 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:16.663416 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:16.663779 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:16.664108 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:16.664506 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.665569 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.666483 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.666961 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.667145 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.667983 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:16.667988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:16.668000 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:16.668363 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:16.668439 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:16.668470 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:16.668583 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:16.668724 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 00:42:16.669021 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260927 00:42:16.670212 9683 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260927 00:42:16.670524 9683 leader_election.cc:302] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba52ad0d679841b49b55516affc2622b; no voters:
I20260927 00:42:16.670933 9683 leader_election.cc:288] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:42:16.670961 9688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:16.671267 9688 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [term 1 LEADER]: Becoming Leader. State: Replica: ba52ad0d679841b49b55516affc2622b, State: Running, Role: LEADER
I20260927 00:42:16.671612 9688 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } }
I20260927 00:42:16.672286 9683 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:16.673266 9689 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba52ad0d679841b49b55516affc2622b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } } }
I20260927 00:42:16.673384 9689 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:16.673738 9697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:16.673725 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba52ad0d679841b49b55516affc2622b. Latest consensus state: current_term: 1 leader_uuid: "ba52ad0d679841b49b55516affc2622b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba52ad0d679841b49b55516affc2622b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44775 } } }
I20260927 00:42:16.674561 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:16.676638 9697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:16.681684 9697 catalog_manager.cc:1347] Generated new cluster ID: 5a270c4fd9fe4bcc9d2db3868d06f659
I20260927 00:42:16.681757 9697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:16.705569 9697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:42:16.706439 9697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:16.714107 9697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba52ad0d679841b49b55516affc2622b: Generated new TSK 0
I20260927 00:42:16.714775 9697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:42:16.721580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--builtin_ntp_servers=127.0.98.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 00:42:16.830154 9707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:16.830392 9707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:16.830428 9707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:16.834393 9707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:16.834439 9707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:16.834520 9707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:16.839205 9707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:16.840195 9707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:16.841601 9707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:16.851007 9715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:16.851034 9712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:16.851054 9713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:16.851171 9707 server_base.cc:1034] running on GCE node
I20260927 00:42:16.851697 9707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:16.852397 9707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:16.853613 9707 hybrid_clock.cc:648] HybridClock initialized: now 1790469736853594 us; error 27 us; skew 500 ppm
I20260927 00:42:16.855597 9707 webserver.cc:460] Webserver started at http://127.0.98.1:35335/ using document root <none> and password file <none>
I20260927 00:42:16.856195 9707 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:16.856290 9707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:16.856531 9707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:16.858729 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0d0165cc73643d8ab1960b06da06d1f"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "a87df353215e8cdf562a12bc2b39a9d9"
server_key_iv: "84622692e0e32ba4aadccf43818d20ed"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.859366 9707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0d0165cc73643d8ab1960b06da06d1f"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "a87df353215e8cdf562a12bc2b39a9d9"
server_key_iv: "84622692e0e32ba4aadccf43818d20ed"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.863420 9707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:42:16.866153 9721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:16.867524 9707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:42:16.867657 9707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "a0d0165cc73643d8ab1960b06da06d1f"
format_stamp: "Formatted at 2026-09-27 00:42:16 on dist-test-slave-pkws"
server_key: "a87df353215e8cdf562a12bc2b39a9d9"
server_key_iv: "84622692e0e32ba4aadccf43818d20ed"
server_key_version: "encryptionkey@0"
I20260927 00:42:16.867784 9707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:16.881973 9707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:16.882592 9707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:16.882788 9707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:16.883334 9707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:16.884210 9707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:16.884284 9707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:16.884356 9707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:16.884397 9707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:16.892866 9707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41961
I20260927 00:42:16.892935 9834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41961 every 8 connection(s)
I20260927 00:42:16.893849 9707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:16.897425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9707
I20260927 00:42:16.897539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:16.897843 392 external_mini_cluster.cc:1428] Setting key 8257d9790b74a6f57c003896011383f3
I20260927 00:42:16.900048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--builtin_ntp_servers=127.0.98.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:42:16.904451 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:16.895768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39395 (local address 127.0.98.62:44775)
0927 00:42:16.895912 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:16.895915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:16.896963 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 00:42:16.898845 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:16.898848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:16.898904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:16.899012 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:16.900189 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.900802 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.902175 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.902599 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.902725 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.903854 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:16.903861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:16.903866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:16.903883 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:16.903912 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:16.903927 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:16.904024 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:16.904160 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 00:42:16.904235 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:42:16.905289 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:16.896134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44775 (local address 127.0.98.1:39395)
0927 00:42:16.896780 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:16.896820 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:16.898597 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:16.899198 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:16.899205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:16.899292 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:16.899988 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.900010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.900960 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.900965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.902042 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:16.902051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:16.902754 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:16.902756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:16.902903 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:16.903561 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:16.903565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:16.903626 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:16.903692 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:16.904021 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:16.904027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:16.904032 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:42:16.904127 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:42:16.904365 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260927 00:42:16.906559 9835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44775
I20260927 00:42:16.906790 9835 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:16.907315 9835 heartbeater.cc:511] Master 127.0.98.62:44775 requested a full tablet report, sending...
I20260927 00:42:16.908767 9647 ts_manager.cc:194] Registered new tserver with Master: a0d0165cc73643d8ab1960b06da06d1f (127.0.98.1:41961)
I20260927 00:42:16.909986 9647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39395
W20260927 00:42:17.014780 9838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:17.015025 9838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:17.015095 9838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:17.019059 9838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:17.019109 9838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:17.019161 9838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:17.024009 9838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:17.025089 9838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:17.026413 9838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:17.035802 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.035941 9838 server_base.cc:1034] running on GCE node
W20260927 00:42:17.035802 9846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.035950 9844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.036661 9838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:17.037312 9838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:17.038498 9838 hybrid_clock.cc:648] HybridClock initialized: now 1790469737038463 us; error 49 us; skew 500 ppm
I20260927 00:42:17.040269 9838 webserver.cc:460] Webserver started at http://127.0.98.2:42633/ using document root <none> and password file <none>
I20260927 00:42:17.040849 9838 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:17.040980 9838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:17.041199 9838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:17.043198 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "91daea859a464bd39cd06cc34e6a68e3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "dc3c9b509c034245d6f36bdc744809bf"
server_key_iv: "63af87bee368548a6e2a5b13119a27e1"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.043757 9838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "91daea859a464bd39cd06cc34e6a68e3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "dc3c9b509c034245d6f36bdc744809bf"
server_key_iv: "63af87bee368548a6e2a5b13119a27e1"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.047555 9838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:42:17.050179 9852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.051373 9838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:42:17.051520 9838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "91daea859a464bd39cd06cc34e6a68e3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "dc3c9b509c034245d6f36bdc744809bf"
server_key_iv: "63af87bee368548a6e2a5b13119a27e1"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.051628 9838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:17.065196 9838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:17.065819 9838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:17.066030 9838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:17.066597 9838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:17.067534 9838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:17.067607 9838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.067678 9838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:17.067721 9838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.076661 9838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37509
I20260927 00:42:17.076809 9965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37509 every 8 connection(s)
I20260927 00:42:17.077566 9838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:17.086472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9838
I20260927 00:42:17.086591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:17.086858 392 external_mini_cluster.cc:1428] Setting key f616b17ab629686ffcd941f65e622395
I20260927 00:42:17.087023 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.079304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32919 (local address 127.0.98.62:44775)
0927 00:42:17.079442 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.079445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.080542 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 00:42:17.082099 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.082103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.082164 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.082257 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.083178 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.083779 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.085056 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.085312 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.085440 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:17.086497 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:17.086501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:17.086504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:17.086517 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:17.086542 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:17.086554 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:17.086629 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:17.086810 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:42:17.086873 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:42:17.087970 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.079681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44775 (local address 127.0.98.2:32919)
0927 00:42:17.080372 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:17.080414 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:17.081869 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:17.082440 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:17.082446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:17.082536 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:17.083020 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:17.083032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.083924 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.083930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:17.084867 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:17.084877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.085475 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.085480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:17.085659 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:17.086202 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:17.086204 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:17.086244 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:17.086334 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:17.086630 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:17.086643 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:17.086653 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 00:42:17.086799 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:42:17.087071 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260927 00:42:17.088937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--builtin_ntp_servers=127.0.98.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 00:42:17.089219 9966 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44775
I20260927 00:42:17.089473 9966 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:17.089977 9966 heartbeater.cc:511] Master 127.0.98.62:44775 requested a full tablet report, sending...
I20260927 00:42:17.091115 9647 ts_manager.cc:194] Registered new tserver with Master: 91daea859a464bd39cd06cc34e6a68e3 (127.0.98.2:37509)
I20260927 00:42:17.091876 9647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32919
W20260927 00:42:17.197511 9969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:17.197746 9969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:17.197782 9969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:17.201737 9969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:17.201784 9969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:17.201834 9969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:17.206440 9969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:17.207396 9969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:17.208745 9969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:17.217728 9974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.217751 9977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.217741 9975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.218083 9969 server_base.cc:1034] running on GCE node
I20260927 00:42:17.218437 9969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:17.219023 9969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:17.220199 9969 hybrid_clock.cc:648] HybridClock initialized: now 1790469737220183 us; error 26 us; skew 500 ppm
I20260927 00:42:17.221994 9969 webserver.cc:460] Webserver started at http://127.0.98.3:45005/ using document root <none> and password file <none>
I20260927 00:42:17.222536 9969 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:17.222625 9969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:17.222851 9969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:17.224754 9969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "0bee30d1a4eb4a9382cde539ed2cfdcd"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "90de8d4a19950fbe4ef542fdd3e3a84c"
server_key_iv: "1a85186771b61df39472ea49b31b4060"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.225351 9969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0bee30d1a4eb4a9382cde539ed2cfdcd"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "90de8d4a19950fbe4ef542fdd3e3a84c"
server_key_iv: "1a85186771b61df39472ea49b31b4060"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.228806 9969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:17.231148 9983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.232256 9969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:17.232383 9969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "0bee30d1a4eb4a9382cde539ed2cfdcd"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "90de8d4a19950fbe4ef542fdd3e3a84c"
server_key_iv: "1a85186771b61df39472ea49b31b4060"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.232488 9969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:17.246939 9969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:17.247542 9969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:17.247726 9969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:17.248253 9969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:17.249208 9969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:17.249287 9969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.249359 9969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:17.249394 9969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.258014 9969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35901
I20260927 00:42:17.258044 10096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35901 every 8 connection(s)
I20260927 00:42:17.258843 9969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:17.265333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 9969
I20260927 00:42:17.265453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:17.265714 392 external_mini_cluster.cc:1428] Setting key baf4a76033bf259464df68d7f9c98266
I20260927 00:42:17.268246 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.260576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57897 (local address 127.0.98.62:44775)
0927 00:42:17.260779 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.260783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.261869 (+ 1086us) server_negotiation.cc:407] Connection header received
0927 00:42:17.263180 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.263182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.263234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.263311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.264141 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.264681 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.266147 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.266453 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.266592 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:17.267806 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:17.267810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:17.267814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:17.267831 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:17.267851 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:17.267860 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:17.267932 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:17.268089 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 00:42:17.268137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:42:17.269081 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.261101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44775 (local address 127.0.98.3:57897)
0927 00:42:17.261731 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:17.261768 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:17.262979 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:17.263462 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:17.263468 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:17.263544 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:17.263999 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:17.264010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.264789 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.264791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:17.266034 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:17.266043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:17.266635 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:17.266639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:17.266839 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:17.267502 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:17.267503 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:17.267538 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:17.267628 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:17.267963 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:17.267970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:17.267974 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:42:17.268069 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:42:17.268266 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260927 00:42:17.270102 10097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44775
I20260927 00:42:17.270316 10097 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:17.270776 10097 heartbeater.cc:511] Master 127.0.98.62:44775 requested a full tablet report, sending...
I20260927 00:42:17.271917 9647 ts_manager.cc:194] Registered new tserver with Master: 0bee30d1a4eb4a9382cde539ed2cfdcd (127.0.98.3:35901)
I20260927 00:42:17.272536 9647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57897
I20260927 00:42:17.279496 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:42:17.282130 9686 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:17.281330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.0.98.62:44775)
0927 00:42:17.281440 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.281444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.281622 (+ 178us) server_negotiation.cc:407] Connection header received
0927 00:42:17.281719 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.281721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.281756 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.281820 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.282002 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47940: BlockingRecv error: recv got EOF from 127.0.0.1:47940 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:42:17.282728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9707
I20260927 00:42:17.288143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9838
I20260927 00:42:17.293282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9969
I20260927 00:42:17.298451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 9616
2026-09-27T00:42:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (851 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T00:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:17Z Disabled control of system clock
I20260927 00:42:17.326776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:42:17.434121 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:17.434408 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:17.434453 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:17.438369 10108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:17.438426 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:17.438442 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:17.438458 10108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:17.438473 10108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:17.443089 10108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:17.444056 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:17.445369 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:17.453722 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.453763 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.453773 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.454052 10108 server_base.cc:1034] running on GCE node
I20260927 00:42:17.454594 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:17.455659 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:17.456866 10108 hybrid_clock.cc:648] HybridClock initialized: now 1790469737456851 us; error 27 us; skew 500 ppm
I20260927 00:42:17.458626 10108 webserver.cc:460] Webserver started at http://127.0.98.62:41717/ using document root <none> and password file <none>
I20260927 00:42:17.459255 10108 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:17.459352 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:17.459578 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:17.461653 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "2f5e154ab1534ddead597b6d8eb2d2d3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "3e994266ebf34d8569bf218f0850217b"
server_key_iv: "b1960da8253ecaff8bed5569e7cc92bc"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.462222 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f5e154ab1534ddead597b6d8eb2d2d3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "3e994266ebf34d8569bf218f0850217b"
server_key_iv: "b1960da8253ecaff8bed5569e7cc92bc"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.465832 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:42:17.468119 10122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.469429 10108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:42:17.469574 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "2f5e154ab1534ddead597b6d8eb2d2d3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "3e994266ebf34d8569bf218f0850217b"
server_key_iv: "b1960da8253ecaff8bed5569e7cc92bc"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.469686 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:17.474097 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:17.474656 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:17.474835 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:17.481326 10108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40187
I20260927 00:42:17.481353 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40187 every 8 connection(s)
I20260927 00:42:17.482173 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:17.485122 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:17.490250 10175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: Bootstrap starting.
I20260927 00:42:17.491851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10108
I20260927 00:42:17.491963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:42:17.492254 392 external_mini_cluster.cc:1428] Setting key 14b3684cc1d967af43950ba5227a0b51
I20260927 00:42:17.492875 10175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:17.493783 10175 log.cc:824] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:17.495858 10175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: No bootstrap required, opened a new log
I20260927 00:42:17.497718 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.493443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.0.98.62:40187)
0927 00:42:17.494061 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.494087 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.494130 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:42:17.494878 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.494900 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.495234 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.495609 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.496278 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:17.496284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:17.496297 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:17.496308 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:17.496367 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:17.496388 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:17.496503 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:17.496608 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 00:42:17.496835 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260927 00:42:17.500043 10175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260927 00:42:17.500232 10175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:17.500270 10175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5e154ab1534ddead597b6d8eb2d2d3, State: Initialized, Role: FOLLOWER
I20260927 00:42:17.500588 10175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260927 00:42:17.500691 10175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:17.500727 10175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:17.500790 10175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:17.501572 10175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260927 00:42:17.501861 10175 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5e154ab1534ddead597b6d8eb2d2d3; no voters:
I20260927 00:42:17.502092 10175 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:42:17.502218 10180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:17.502553 10180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2f5e154ab1534ddead597b6d8eb2d2d3, State: Running, Role: LEADER
I20260927 00:42:17.502897 10180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } }
I20260927 00:42:17.503203 10175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:17.504315 10182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5e154ab1534ddead597b6d8eb2d2d3. Latest consensus state: current_term: 1 leader_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260927 00:42:17.504513 10182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:17.504861 10181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5e154ab1534ddead597b6d8eb2d2d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40187 } } }
I20260927 00:42:17.504994 10181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3 [sys.catalog]: This master's current role is: LEADER
W20260927 00:42:17.507542 10195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:42:17.507629 10195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:42:17.507695 10190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:17.508339 10190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:17.513125 10190 catalog_manager.cc:1347] Generated new cluster ID: 7a9d224ae8db431fa6a074f42ad1f357
I20260927 00:42:17.513185 10190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:17.524273 10190 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:42:17.524569 10190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:17.544162 10190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5e154ab1534ddead597b6d8eb2d2d3: Generated new TSK 0
I20260927 00:42:17.544929 10190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:42:17.554629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--builtin_ntp_servers=127.0.98.20:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 00:42:17.661372 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:17.661615 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:17.661652 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:17.665517 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:17.665562 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:17.665618 10199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:17.670210 10199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:17.671097 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:17.672318 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:17.681090 10205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.681090 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.681363 10199 server_base.cc:1034] running on GCE node
W20260927 00:42:17.681135 10207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.681865 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:17.682492 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:17.683681 10199 hybrid_clock.cc:648] HybridClock initialized: now 1790469737683665 us; error 33 us; skew 500 ppm
I20260927 00:42:17.685550 10199 webserver.cc:460] Webserver started at http://127.0.98.1:42393/ using document root <none> and password file <none>
I20260927 00:42:17.686091 10199 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:17.686203 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:17.686416 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:17.688310 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d65b52351224945bab184e0925a8087"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "96f4acf72bad501a140c3622453fd2e6"
server_key_iv: "9cb6738340978fb0a98437edef9a13f9"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.688853 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d65b52351224945bab184e0925a8087"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "96f4acf72bad501a140c3622453fd2e6"
server_key_iv: "9cb6738340978fb0a98437edef9a13f9"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.692350 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:42:17.694787 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.695995 10199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:17.696134 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "4d65b52351224945bab184e0925a8087"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "96f4acf72bad501a140c3622453fd2e6"
server_key_iv: "9cb6738340978fb0a98437edef9a13f9"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.696257 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:17.700937 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:17.701542 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:17.701772 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:17.702306 10199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:17.703168 10199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:17.703239 10199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.703301 10199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:17.703353 10199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.712124 10199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33735
I20260927 00:42:17.712150 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33735 every 8 connection(s)
I20260927 00:42:17.713085 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:17.718788 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.714762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42811 (local address 127.0.98.62:40187)
0927 00:42:17.714875 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.714879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.715937 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 00:42:17.717222 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.717225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.717263 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.717341 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.718427 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:17.718430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:17.718431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:17.718463 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:17.718479 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:17.718486 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:17.718557 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:17.718652 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 00:42:17.718687 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:42:17.719612 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.715094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.1:42811)
0927 00:42:17.715790 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:17.715828 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:17.717024 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:17.717498 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:17.717504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:17.717586 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:17.718190 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:17.718191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:17.718230 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:17.718292 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:17.718541 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:17.718547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:17.718550 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:42:17.718665 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:42:17.718870 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":117,"threads_started":1}
I20260927 00:42:17.720525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10199
I20260927 00:42:17.720634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:17.720956 10327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260927 00:42:17.720985 392 external_mini_cluster.cc:1428] Setting key bcde86dd01877a303e261c086f15f8cc
I20260927 00:42:17.721186 10327 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:17.721596 10327 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260927 00:42:17.722988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--builtin_ntp_servers=127.0.98.20:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:42:17.723158 10139 ts_manager.cc:194] Registered new tserver with Master: 4d65b52351224945bab184e0925a8087 (127.0.98.1:33735)
W20260927 00:42:17.836227 10330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:17.836488 10330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:17.836531 10330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:17.840742 10330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:17.840806 10330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:17.840865 10330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:17.845785 10330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:17.846817 10330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:17.848156 10330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:17.857751 10338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.857787 10336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:17.857751 10335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:17.858028 10330 server_base.cc:1034] running on GCE node
I20260927 00:42:17.858542 10330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:17.859160 10330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:17.860369 10330 hybrid_clock.cc:648] HybridClock initialized: now 1790469737860356 us; error 38 us; skew 500 ppm
I20260927 00:42:17.862295 10330 webserver.cc:460] Webserver started at http://127.0.98.2:37039/ using document root <none> and password file <none>
I20260927 00:42:17.862869 10330 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:17.862962 10330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:17.863186 10330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:17.865258 10330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "4738c75068cf4d049ded3b8830de9cf3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "d7680085a78784e3b944b369117e58f9"
server_key_iv: "77618a0a1218268aa424fcd39d00845c"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.865836 10330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4738c75068cf4d049ded3b8830de9cf3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "d7680085a78784e3b944b369117e58f9"
server_key_iv: "77618a0a1218268aa424fcd39d00845c"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.869501 10330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:42:17.871760 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.873029 10330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:42:17.873147 10330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "4738c75068cf4d049ded3b8830de9cf3"
format_stamp: "Formatted at 2026-09-27 00:42:17 on dist-test-slave-pkws"
server_key: "d7680085a78784e3b944b369117e58f9"
server_key_iv: "77618a0a1218268aa424fcd39d00845c"
server_key_version: "encryptionkey@0"
I20260927 00:42:17.873270 10330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:17.877671 10330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:17.878238 10330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:17.878366 10330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:17.878829 10330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:17.879658 10330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:17.879704 10330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.879769 10330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:17.879804 10330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:17.887975 10330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40823
I20260927 00:42:17.888001 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40823 every 8 connection(s)
I20260927 00:42:17.888774 10330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:17.889217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10330
I20260927 00:42:17.889333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:17.889668 392 external_mini_cluster.cc:1428] Setting key fd422aaf8dadaec9936e99433b5472d3
I20260927 00:42:17.891736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--builtin_ntp_servers=127.0.98.20:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 00:42:17.895704 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.890577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45605 (local address 127.0.98.62:40187)
0927 00:42:17.890718 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:17.890736 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:17.892075 (+ 1339us) server_negotiation.cc:407] Connection header received
0927 00:42:17.893840 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:17.893843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:17.893881 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:17.893959 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:17.895059 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:17.895062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:17.895064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:17.895075 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:17.895095 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:17.895102 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:17.895180 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:17.895379 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 00:42:17.895419 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:42:17.896651 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:17.891141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.2:45605)
0927 00:42:17.891925 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:17.891977 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:17.893641 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:17.894107 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:17.894114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:17.894199 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:17.894761 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:17.894766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:17.894819 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:17.894904 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:17.895193 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:17.895203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:17.895212 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:42:17.895378 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 00:42:17.895621 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260927 00:42:17.897887 10458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260927 00:42:17.898128 10458 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:17.898631 10458 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260927 00:42:17.899732 10139 ts_manager.cc:194] Registered new tserver with Master: 4738c75068cf4d049ded3b8830de9cf3 (127.0.98.2:40823)
W20260927 00:42:18.002497 10461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:18.002745 10461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:18.002782 10461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:18.007170 10461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:18.007225 10461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:18.007315 10461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:18.011996 10461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:18.013080 10461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:18.014420 10461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:18.023765 10466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:18.023774 10467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:18.023777 10469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:18.024144 10461 server_base.cc:1034] running on GCE node
I20260927 00:42:18.024585 10461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:18.025228 10461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:18.026415 10461 hybrid_clock.cc:648] HybridClock initialized: now 1790469738026405 us; error 33 us; skew 500 ppm
I20260927 00:42:18.028105 10461 webserver.cc:460] Webserver started at http://127.0.98.3:41979/ using document root <none> and password file <none>
I20260927 00:42:18.028674 10461 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:18.028764 10461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:18.029039 10461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:18.031050 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "04cfe33c19814875ac6f3d264cccdb0d"
format_stamp: "Formatted at 2026-09-27 00:42:18 on dist-test-slave-pkws"
server_key: "8f492ac1d8fc6187056acfcdb3548828"
server_key_iv: "ab10e449b5bf2ab65fd14b8155c4b9d3"
server_key_version: "encryptionkey@0"
I20260927 00:42:18.031625 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "04cfe33c19814875ac6f3d264cccdb0d"
format_stamp: "Formatted at 2026-09-27 00:42:18 on dist-test-slave-pkws"
server_key: "8f492ac1d8fc6187056acfcdb3548828"
server_key_iv: "ab10e449b5bf2ab65fd14b8155c4b9d3"
server_key_version: "encryptionkey@0"
I20260927 00:42:18.035120 10461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 00:42:18.037456 10475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:18.038657 10461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:42:18.038792 10461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "04cfe33c19814875ac6f3d264cccdb0d"
format_stamp: "Formatted at 2026-09-27 00:42:18 on dist-test-slave-pkws"
server_key: "8f492ac1d8fc6187056acfcdb3548828"
server_key_iv: "ab10e449b5bf2ab65fd14b8155c4b9d3"
server_key_version: "encryptionkey@0"
I20260927 00:42:18.038904 10461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:18.043324 10461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:18.043895 10461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:18.044077 10461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:18.044605 10461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:18.045534 10461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:18.045603 10461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:18.045667 10461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:18.045714 10461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:18.054833 10461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34797
I20260927 00:42:18.054860 10588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34797 every 8 connection(s)
I20260927 00:42:18.055677 10461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:18.058432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10461
I20260927 00:42:18.058549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:18.058905 392 external_mini_cluster.cc:1428] Setting key a56300ebf2d64bad2f40e5e7997ea202
I20260927 00:42:18.062202 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:18.057421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58355 (local address 127.0.98.62:40187)
0927 00:42:18.057574 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:18.057578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:18.058820 (+ 1242us) server_negotiation.cc:407] Connection header received
0927 00:42:18.060472 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:18.060475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:18.060514 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:18.060601 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:18.061849 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:18.061852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:18.061854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:42:18.061865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:18.061891 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:42:18.061899 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:18.061968 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:18.062065 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 00:42:18.062101 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:42:18.063041 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:18.057796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40187 (local address 127.0.98.3:58355)
0927 00:42:18.058609 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:18.058663 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:18.060253 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:18.060773 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:18.060779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:18.060920 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:18.061593 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:42:18.061595 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:18.061643 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:42:18.061718 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:18.061957 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:18.061961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:18.061964 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 00:42:18.062079 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:42:18.062258 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260927 00:42:18.064026 10589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40187
I20260927 00:42:18.064248 10589 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:18.064621 10589 heartbeater.cc:511] Master 127.0.98.62:40187 requested a full tablet report, sending...
I20260927 00:42:18.065645 10139 ts_manager.cc:194] Registered new tserver with Master: 04cfe33c19814875ac6f3d264cccdb0d (127.0.98.3:34797)
I20260927 00:42:18.072680 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 00:42:18.075644 10178 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:42:18.074758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.0.98.62:40187)
0927 00:42:18.074905 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:18.074909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:18.075070 (+ 161us) server_negotiation.cc:407] Connection header received
0927 00:42:18.075212 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:18.075215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:18.075254 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:18.075336 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:18.075555 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53494: BlockingRecv error: recv got EOF from 127.0.0.1:53494 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:42:18.076125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10199
I20260927 00:42:18.081580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10330
I20260927 00:42:18.086764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10461
I20260927 00:42:18.091777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10108
2026-09-27T00:42:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (793 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:42:18 dist-test-slave-pkws krb5kdc[10599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:42:18 dist-test-slave-pkws krb5kdc[10599](info): set up 2 sockets
Sep 27 00:42:18 dist-test-slave-pkws krb5kdc[10599](info): commencing operation
krb5kdc: starting...
W20260927 00:42:19.145954 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:42:19.264703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:19.372306 10612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.372548 10612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.372589 10612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.376520 10612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:42:19.376573 10612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.376593 10612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.376609 10612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:42:19.376670 10612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:42:19.381951 10612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:19.382937 10612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.383919 10612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.392309 10617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.392316 10620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.392323 10618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.392712 10612 server_base.cc:1034] running on GCE node
I20260927 00:42:19.393401 10612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.394513 10612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.395733 10612 hybrid_clock.cc:648] HybridClock initialized: now 1790469739395716 us; error 30 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.398687 10612 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:42:19.399544 10612 webserver.cc:460] Webserver started at http://127.0.98.62:39239/ using document root <none> and password file <none>
I20260927 00:42:19.400085 10612 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.400142 10612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.400305 10612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:19.402374 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "073cc1f4e56c492fb2754c5013f2fd48"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "335a7d95bd8ced5be7076adf4435cb7a"
server_key_iv: "57daa3088a93318cfa7cfc394a51637e"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.402886 10612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "073cc1f4e56c492fb2754c5013f2fd48"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "335a7d95bd8ced5be7076adf4435cb7a"
server_key_iv: "57daa3088a93318cfa7cfc394a51637e"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.406461 10612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 00:42:19.408681 10627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.410082 10612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:19.410187 10612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "073cc1f4e56c492fb2754c5013f2fd48"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "335a7d95bd8ced5be7076adf4435cb7a"
server_key_iv: "57daa3088a93318cfa7cfc394a51637e"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.410270 10612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.422401 10612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.425151 10612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.425297 10612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.432071 10612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40365
I20260927 00:42:19.432354 10679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40365 every 8 connection(s)
I20260927 00:42:19.433106 10612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:42:19.435833 10680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:19.440325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10612
I20260927 00:42:19.440459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:42:19.440726 392 external_mini_cluster.cc:1428] Setting key 197057bf97a6c771cd2d40f56e1fe150
I20260927 00:42:19.441282 10680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: Bootstrap starting.
I20260927 00:42:19.443671 10680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:19.444388 10680 log.cc:824] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:19.446262 10680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: No bootstrap required, opened a new log
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:19.449741 10680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 00:42:19.449988 10680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:19.450067 10680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073cc1f4e56c492fb2754c5013f2fd48, State: Initialized, Role: FOLLOWER
I20260927 00:42:19.450572 10680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 00:42:19.450745 10680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:42:19.450840 10680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:42:19.450941 10680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:19.451835 10680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 00:42:19.452180 10680 leader_election.cc:302] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 073cc1f4e56c492fb2754c5013f2fd48; no voters:
I20260927 00:42:19.452509 10680 leader_election.cc:288] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:42:19.452572 10685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:19.452771 10685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [term 1 LEADER]: Becoming Leader. State: Replica: 073cc1f4e56c492fb2754c5013f2fd48, State: Running, Role: LEADER
I20260927 00:42:19.453135 10685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 00:42:19.453774 10680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:42:19.454310 10685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 073cc1f4e56c492fb2754c5013f2fd48. Latest consensus state: current_term: 1 leader_uuid: "073cc1f4e56c492fb2754c5013f2fd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } } }
I20260927 00:42:19.454420 10685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:19.454725 10686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "073cc1f4e56c492fb2754c5013f2fd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073cc1f4e56c492fb2754c5013f2fd48" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } } }
I20260927 00:42:19.454819 10686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48 [sys.catalog]: This master's current role is: LEADER
I20260927 00:42:19.455214 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.442171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.62:40365)
0927 00:42:19.442567 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.442598 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.442634 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:42:19.443247 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.443264 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.443603 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.443965 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.444769 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.445863 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.446793 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.447287 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.447460 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.449949 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.449954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.449966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.450014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.451827 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.452270 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.452276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.452281 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.452425 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.452731 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.452736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.452737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.453265 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.453663 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.453825 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.454147 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
W20260927 00:42:19.459712 10700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:42:19.459810 10700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:42:19.459896 10689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:42:19.460589 10689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:42:19.465803 10689 catalog_manager.cc:1347] Generated new cluster ID: 79100be9c1ad45039cdf1deba308b984
I20260927 00:42:19.465862 10689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:42:19.481832 10689 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:42:19.483100 10689 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:42:19.493129 10689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: Generated new TSK 0
I20260927 00:42:19.493867 10689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:42:19.531399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:19.655103 10706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.655354 10706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.655395 10706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.659425 10706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.659477 10706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.659533 10706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:42:19.664453 10706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:19.665423 10706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.666327 10706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.675475 10712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.675469 10714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.675485 10711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.675637 10706 server_base.cc:1034] running on GCE node
I20260927 00:42:19.676101 10706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.676769 10706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.677945 10706 hybrid_clock.cc:648] HybridClock initialized: now 1790469739677920 us; error 43 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.680521 10706 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:42:19.681402 10706 webserver.cc:460] Webserver started at http://127.0.98.1:45553/ using document root <none> and password file <none>
I20260927 00:42:19.682185 10706 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.682303 10706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.682534 10706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:19.684442 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "380815f2d0aa45758635942a23596889"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "6f10df2b3d9ebf947ac7066102fc28d7"
server_key_iv: "4fc44d71ba5793512e73830cb789e77a"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.685048 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "380815f2d0aa45758635942a23596889"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "6f10df2b3d9ebf947ac7066102fc28d7"
server_key_iv: "4fc44d71ba5793512e73830cb789e77a"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.688566 10706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:42:19.690896 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.692101 10706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:42:19.692232 10706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "380815f2d0aa45758635942a23596889"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "6f10df2b3d9ebf947ac7066102fc28d7"
server_key_iv: "4fc44d71ba5793512e73830cb789e77a"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.692337 10706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.705013 10706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.708096 10706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.708276 10706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.708807 10706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:19.709568 10706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:19.709636 10706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.709710 10706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:19.709764 10706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.718605 10706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35261
I20260927 00:42:19.718636 10834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35261 every 8 connection(s)
I20260927 00:42:19.719403 10706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:42:19.728097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10706
I20260927 00:42:19.728195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:42:19.728528 392 external_mini_cluster.cc:1428] Setting key 453af50117b495be50ed2c4b28d602fd
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:19.733397 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.721170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60717 (local address 127.0.98.62:40365)
0927 00:42:19.721294 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.721298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.722084 (+ 786us) server_negotiation.cc:407] Connection header received
0927 00:42:19.723326 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.723329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.723381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.723486 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.724585 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.725179 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.726423 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.726696 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.726799 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.729418 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.729422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.729424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.729448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.731280 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.731749 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.731757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.731762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.731863 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.732292 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.732294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.732296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.732471 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.732554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.733175 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.733277 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:42:19.734222 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.721548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.1:60717)
0927 00:42:19.721950 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.721989 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.723128 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.723683 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.723690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.724013 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.724447 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.724460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.725298 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.725302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.726261 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.726272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.726846 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.726850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.727023 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.727609 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.727624 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.729293 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.731398 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.731402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.731412 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.731647 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.731983 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.731989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.731993 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.732190 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.732577 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.732584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.732880 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.733121 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.733429 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":98,"threads_started":1}
I20260927 00:42:19.735381 10835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
I20260927 00:42:19.735620 10835 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:19.736058 10835 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
I20260927 00:42:19.737524 10644 ts_manager.cc:194] Registered new tserver with Master: 380815f2d0aa45758635942a23596889 (127.0.98.1:35261)
I20260927 00:42:19.738584 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60717
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:42:19.756217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:19.866952 10840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:19.867199 10840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:19.867241 10840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:19.871301 10840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:19.871359 10840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:19.871465 10840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:42:19.876524 10840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:19.877565 10840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:19.878545 10840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:19.887681 10846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.887692 10845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:19.887760 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:19.887988 10840 server_base.cc:1034] running on GCE node
I20260927 00:42:19.888392 10840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:19.889077 10840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:19.890257 10840 hybrid_clock.cc:648] HybridClock initialized: now 1790469739890221 us; error 45 us; skew 500 ppm
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:19.892874 10840 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:42:19.893716 10840 webserver.cc:460] Webserver started at http://127.0.98.2:34021/ using document root <none> and password file <none>
I20260927 00:42:19.894241 10840 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:19.894338 10840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:19.894559 10840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:19.896600 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c3b476860cd4ce09d3b715b167d76c0"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "486ed5f57777b15af3b5cc9531bd6f0e"
server_key_iv: "657d912a50c0192f3267d05fa4d740a2"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.897179 10840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c3b476860cd4ce09d3b715b167d76c0"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "486ed5f57777b15af3b5cc9531bd6f0e"
server_key_iv: "657d912a50c0192f3267d05fa4d740a2"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.900697 10840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 00:42:19.903107 10855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.904269 10840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:42:19.904398 10840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "8c3b476860cd4ce09d3b715b167d76c0"
format_stamp: "Formatted at 2026-09-27 00:42:19 on dist-test-slave-pkws"
server_key: "486ed5f57777b15af3b5cc9531bd6f0e"
server_key_iv: "657d912a50c0192f3267d05fa4d740a2"
server_key_version: "encryptionkey@0"
I20260927 00:42:19.904503 10840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:19.915319 10840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:19.918464 10840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:19.918663 10840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:19.919209 10840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:19.919981 10840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:19.920051 10840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.920115 10840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:19.920163 10840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:19.929071 10840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39501
I20260927 00:42:19.929104 10968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39501 every 8 connection(s)
I20260927 00:42:19.929912 10840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:42:19.931783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10840
I20260927 00:42:19.931911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:42:19.932202 392 external_mini_cluster.cc:1428] Setting key 6244ffdf5d5d9b70d99fe6bf1b974524
Sep 27 00:42:19 dist-test-slave-pkws krb5kdc[10599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:42:19.944401 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.931624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38775 (local address 127.0.98.62:40365)
0927 00:42:19.931767 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:19.931772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:19.932653 (+ 881us) server_negotiation.cc:407] Connection header received
0927 00:42:19.933915 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:19.933919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:19.933974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:19.934073 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:19.935253 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.935764 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.937584 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.937801 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.937883 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.940689 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:19.940699 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:19.940702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:19.940724 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:19.942197 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.942723 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.942725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.942726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.942833 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:19.943203 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:19.943206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:19.943208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:19.943382 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:19.943489 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:19.944071 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 00:42:19.944237 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:42:19.945426 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:19.932065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.2:38775)
0927 00:42:19.932494 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:19.932543 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:19.933712 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:19.934201 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:19.934207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:19.934546 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:19.935097 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.935120 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.935902 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.935907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.937445 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:19.937458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:19.937955 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:19.937960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:19.938174 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:19.938840 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:19.938865 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:19.940526 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:19.942312 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.942318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.942331 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.942582 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.942949 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:19.942952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:19.942953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:19.943099 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:19.943487 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:19.943494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:19.943834 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 00:42:19.944015 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:42:19.944329 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":123,"threads_started":1}
I20260927 00:42:19.946746 10969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
I20260927 00:42:19.947010 10969 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:19.947482 10969 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
I20260927 00:42:19.948444 10644 ts_manager.cc:194] Registered new tserver with Master: 8c3b476860cd4ce09d3b715b167d76c0 (127.0.98.2:39501)
I20260927 00:42:19.949043 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:42:19.960065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:42:20.067560 10974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:42:20.067800 10974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:42:20.067840 10974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:42:20.071683 10974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:42:20.071739 10974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:42:20.071794 10974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:42:20.076774 10974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:42:20.077692 10974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:42:20.078606 10974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:42:20.087776 10982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:42:20.087769 10980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:20.087869 10974 server_base.cc:1034] running on GCE node
W20260927 00:42:20.087978 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:42:20.088438 10974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:42:20.089066 10974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:42:20.090222 10974 hybrid_clock.cc:648] HybridClock initialized: now 1790469740090205 us; error 29 us; skew 500 ppm
Sep 27 00:42:20 dist-test-slave-pkws krb5kdc[10599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:42:20.092935 10974 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:42:20.093714 10974 webserver.cc:460] Webserver started at http://127.0.98.3:46843/ using document root <none> and password file <none>
I20260927 00:42:20.094241 10974 fs_manager.cc:359] Metadata directory not provided
I20260927 00:42:20.094301 10974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:42:20.094502 10974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:42:20.096513 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad"
format_stamp: "Formatted at 2026-09-27 00:42:20 on dist-test-slave-pkws"
server_key: "5d57010480d8b5a1a51caba4b53821b8"
server_key_iv: "fa3bdeb6d5fd0b8327889285b0cf9e4e"
server_key_version: "encryptionkey@0"
I20260927 00:42:20.097100 10974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad"
format_stamp: "Formatted at 2026-09-27 00:42:20 on dist-test-slave-pkws"
server_key: "5d57010480d8b5a1a51caba4b53821b8"
server_key_iv: "fa3bdeb6d5fd0b8327889285b0cf9e4e"
server_key_version: "encryptionkey@0"
I20260927 00:42:20.100421 10974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:42:20.102738 10989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:20.103855 10974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:42:20.103969 10974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad"
format_stamp: "Formatted at 2026-09-27 00:42:20 on dist-test-slave-pkws"
server_key: "5d57010480d8b5a1a51caba4b53821b8"
server_key_iv: "fa3bdeb6d5fd0b8327889285b0cf9e4e"
server_key_version: "encryptionkey@0"
I20260927 00:42:20.104061 10974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:42:20.143620 10974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:42:20.147004 10974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:42:20.147212 10974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:42:20.147802 10974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:42:20.148599 10974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:42:20.148645 10974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:20.148713 10974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:42:20.148761 10974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:42:20.157867 10974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40555
I20260927 00:42:20.157897 11102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40555 every 8 connection(s)
I20260927 00:42:20.158762 10974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:42:20.165848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 10974
I20260927 00:42:20.165952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:42:20.166321 392 external_mini_cluster.cc:1428] Setting key 777d2b2eaaf29f8b8f36818e9f120b92
Sep 27 00:42:20 dist-test-slave-pkws krb5kdc[10599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:42:20.177004 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.160546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56491 (local address 127.0.98.62:40365)
0927 00:42:20.160703 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.160708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.161538 (+ 830us) server_negotiation.cc:407] Connection header received
0927 00:42:20.163223 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.163226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.163281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.163355 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:20.164434 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.165016 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.166251 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.166556 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.166666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.169376 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:20.169379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:20.169381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:20.169416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:20.175176 (+ 5760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:20.175663 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:20.175665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:20.175667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:20.175748 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:20.176034 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:20.176036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:20.176037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:20.176200 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:20.176271 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.176724 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.176823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:42:20.177913 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.160972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.3:56491)
0927 00:42:20.161389 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.161430 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.163033 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.163545 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.163552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.163872 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:42:20.164313 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.164324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.165131 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.165134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.166131 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.166141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.166710 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.166715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.166913 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.167533 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:42:20.167567 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:42:20.169236 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:42:20.175283 (+ 6047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:20.175287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:20.175299 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:20.175577 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:20.175829 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:42:20.175832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:42:20.175833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:42:20.175957 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:42:20.176283 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:42:20.176288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:42:20.176498 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.176678 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.176988 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20260927 00:42:20.179068 11103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
I20260927 00:42:20.179309 11103 heartbeater.cc:464] Registering TS with master...
I20260927 00:42:20.179848 11103 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
I20260927 00:42:20.180838 10644 ts_manager.cc:194] Registered new tserver with Master: 19f8eb5cd0df4ed9a8cdd3b57a3508ad (127.0.98.3:40555)
I20260927 00:42:20.181681 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56491
I20260927 00:42:20.191675 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:42:20.200229 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.193660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.98.62:40365)
0927 00:42:20.193784 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.193788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.193965 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:42:20.194056 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.194058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.194092 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.194162 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:42:20.194809 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.195292 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.196145 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.196357 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.196435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.197682 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:42:20.197685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:42:20.197687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:42:20.197707 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:42:20.198915 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:20.199286 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:20.199288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:20.199289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:20.199358 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:42:20.199583 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:42:20.199585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:42:20.199586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:42:20.199772 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:42:20.199871 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.200046 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.200129 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:42:20.202750 10644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:42:20.205899 10644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:42:20.219862 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.215078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.0.98.1:35261)
0927 00:42:20.215322 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.215336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.215356 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:42:20.215485 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.215490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.215666 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.215810 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.216062 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.217253 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.218669 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219202 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.219322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.219459 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.219634 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.219720 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":56,"threads_started":1}
I20260927 00:42:20.219851 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.214237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.0.1:34152)
0927 00:42:20.214462 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.214477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.214553 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.215808 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.215811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.215830 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.215930 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.215940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.217391 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.217395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.218557 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.218564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219322 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.219325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.219435 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.219471 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.219606 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.219718 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260927 00:42:20.220150 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.214744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.98.2:39501)
0927 00:42:20.215011 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.215021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.215044 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:42:20.215105 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.215109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.215315 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.215474 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.216019 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.216830 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.218433 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219250 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.219351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.219476 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.219919 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.220015 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":101,"threads_started":1}
I20260927 00:42:20.220139 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.213719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40555 (local address 127.0.0.1:55026)
0927 00:42:20.214036 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.214086 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.214208 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.216019 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.216028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.216045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.216129 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.216135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.217435 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.217440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.218800 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.218810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219814 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.219817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.219898 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.219917 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.219963 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.220017 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":223,"thread_start_us":87,"threads_started":1}
I20260927 00:42:20.220325 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.214553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39501 (local address 127.0.0.1:45698)
0927 00:42:20.214797 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.214810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.214893 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.215692 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.215698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.215723 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.215858 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.215865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.216996 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.216999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.218305 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.218315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219542 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.219547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.219633 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.219656 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.219748 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.219829 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":26,"thread_start_us":89,"threads_started":1}
I20260927 00:42:20.220816 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.213871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.3:40555)
0927 00:42:20.215591 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.215604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.215627 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:42:20.215693 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.215697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.215875 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.216029 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.216219 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.217329 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.218932 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.219693 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.220457 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.220585 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.220636 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.220712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621,"thread_start_us":95,"threads_started":1}
I20260927 00:42:20.223743 10903 tablet_service.cc:1467] Processing CreateTablet for tablet 08e713ee1bb540538db704edecbfd511 (DEFAULT_TABLE table=test-table [id=1b6e56f5fe3a474592305c247f1f007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:20.223744 10769 tablet_service.cc:1467] Processing CreateTablet for tablet 08e713ee1bb540538db704edecbfd511 (DEFAULT_TABLE table=test-table [id=1b6e56f5fe3a474592305c247f1f007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:20.224845 10903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e713ee1bb540538db704edecbfd511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:20.224845 10769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e713ee1bb540538db704edecbfd511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:20.226455 11037 tablet_service.cc:1467] Processing CreateTablet for tablet 08e713ee1bb540538db704edecbfd511 (DEFAULT_TABLE table=test-table [id=1b6e56f5fe3a474592305c247f1f007d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:42:20.227222 11037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e713ee1bb540538db704edecbfd511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:42:20.231990 11121 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: Bootstrap starting.
I20260927 00:42:20.232393 11120 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Bootstrap starting.
I20260927 00:42:20.232869 11122 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Bootstrap starting.
I20260927 00:42:20.234905 11122 tablet_bootstrap.cc:654] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:20.234905 11121 tablet_bootstrap.cc:654] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:20.235062 11120 tablet_bootstrap.cc:654] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Neither blocks nor log segments found. Creating new log.
I20260927 00:42:20.235734 11120 log.cc:824] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:20.235788 11121 log.cc:824] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:20.236016 11122 log.cc:824] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Log is configured to *not* fsync() on all Append() calls
I20260927 00:42:20.237517 11120 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: No bootstrap required, opened a new log
I20260927 00:42:20.237740 11120 ts_tablet_manager.cc:1397] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:42:20.237963 11122 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: No bootstrap required, opened a new log
I20260927 00:42:20.237963 11121 tablet_bootstrap.cc:492] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: No bootstrap required, opened a new log
I20260927 00:42:20.238144 11121 ts_tablet_manager.cc:1397] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260927 00:42:20.238144 11122 ts_tablet_manager.cc:1397] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 00:42:20.240875 11120 raft_consensus.cc:348] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.241175 11120 raft_consensus.cc:374] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:20.241274 11120 raft_consensus.cc:729] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3b476860cd4ce09d3b715b167d76c0, State: Initialized, Role: FOLLOWER
I20260927 00:42:20.241734 11120 consensus_queue.cc:260] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.241904 11122 raft_consensus.cc:348] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.242334 10969 heartbeater.cc:503] Master 127.0.98.62:40365 was elected leader, sending a full tablet report...
I20260927 00:42:20.242244 11121 raft_consensus.cc:348] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.242496 11121 raft_consensus.cc:374] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:20.242654 11121 raft_consensus.cc:729] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380815f2d0aa45758635942a23596889, State: Initialized, Role: FOLLOWER
I20260927 00:42:20.242720 11120 ts_tablet_manager.cc:1428] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 00:42:20.242823 11122 raft_consensus.cc:374] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:42:20.242913 11122 raft_consensus.cc:729] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f8eb5cd0df4ed9a8cdd3b57a3508ad, State: Initialized, Role: FOLLOWER
I20260927 00:42:20.243103 11121 consensus_queue.cc:260] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.243382 11122 consensus_queue.cc:260] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.243726 10835 heartbeater.cc:503] Master 127.0.98.62:40365 was elected leader, sending a full tablet report...
I20260927 00:42:20.244055 11121 ts_tablet_manager.cc:1428] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 00:42:20.244297 11122 ts_tablet_manager.cc:1428] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:42:20.244632 11103 heartbeater.cc:503] Master 127.0.98.62:40365 was elected leader, sending a full tablet report...
I20260927 00:42:20.285691 11128 raft_consensus.cc:482] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:20.285892 11128 raft_consensus.cc:504] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.287088 11128 leader_election.cc:288] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 380815f2d0aa45758635942a23596889 (127.0.98.1:35261), 8c3b476860cd4ce09d3b715b167d76c0 (127.0.98.2:39501)
I20260927 00:42:20.292277 11127 raft_consensus.cc:482] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:42:20.292498 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.287718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45397 (local address 127.0.98.2:39501)
0927 00:42:20.287869 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.287873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.288003 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:42:20.288115 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.288118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.288163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.288245 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.288493 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.289287 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291122 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.291736 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291847 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.291910 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.292285 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.292370 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:42:20.292516 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.287285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.98.3:51637)
0927 00:42:20.287572 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.287589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.287685 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.288124 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.288127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.288144 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.288274 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.288280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.289404 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.289408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.290982 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.290993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.291884 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.292140 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.292215 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.292280 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.292385 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":323}
I20260927 00:42:20.292725 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.287404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51637 (local address 127.0.98.1:35261)
0927 00:42:20.287645 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.287651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.287667 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:42:20.287933 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.287936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.288004 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.288093 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.288390 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.289286 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291122 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.291742 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291847 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.291926 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.292403 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.292476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260927 00:42:20.292582 11127 raft_consensus.cc:504] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.292939 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.287607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39501 (local address 127.0.98.3:45397)
0927 00:42:20.287914 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.287926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.288013 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.288266 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.288268 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.288280 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.288398 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.288404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.289404 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.289408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.290990 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.290995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.291884 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.291888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.292126 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.292172 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.292265 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.292764 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":233,"thread_start_us":140,"threads_started":1}
I20260927 00:42:20.293334 10789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380815f2d0aa45758635942a23596889" is_pre_election: true
I20260927 00:42:20.293491 10923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3b476860cd4ce09d3b715b167d76c0" is_pre_election: true
I20260927 00:42:20.293588 10789 raft_consensus.cc:2454] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f8eb5cd0df4ed9a8cdd3b57a3508ad in term 0.
I20260927 00:42:20.293726 10923 raft_consensus.cc:2454] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19f8eb5cd0df4ed9a8cdd3b57a3508ad in term 0.
I20260927 00:42:20.294219 10990 leader_election.cc:302] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f8eb5cd0df4ed9a8cdd3b57a3508ad, 8c3b476860cd4ce09d3b715b167d76c0; no voters:
I20260927 00:42:20.294268 11127 leader_election.cc:288] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f8eb5cd0df4ed9a8cdd3b57a3508ad (127.0.98.3:40555), 8c3b476860cd4ce09d3b715b167d76c0 (127.0.98.2:39501)
I20260927 00:42:20.294844 11128 raft_consensus.cc:2781] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:42:20.294902 11128 raft_consensus.cc:482] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:42:20.294929 11128 raft_consensus.cc:3037] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.296263 11128 raft_consensus.cc:504] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.296599 11128 leader_election.cc:288] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [CANDIDATE]: Term 1 election: Requested vote from peers 380815f2d0aa45758635942a23596889 (127.0.98.1:35261), 8c3b476860cd4ce09d3b715b167d76c0 (127.0.98.2:39501)
I20260927 00:42:20.297029 10923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3b476860cd4ce09d3b715b167d76c0"
I20260927 00:42:20.297066 10789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380815f2d0aa45758635942a23596889"
I20260927 00:42:20.297148 10923 raft_consensus.cc:3037] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.297170 10789 raft_consensus.cc:3037] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:42:20.298391 10923 raft_consensus.cc:2454] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f8eb5cd0df4ed9a8cdd3b57a3508ad in term 1.
I20260927 00:42:20.298444 10789 raft_consensus.cc:2454] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19f8eb5cd0df4ed9a8cdd3b57a3508ad in term 1.
I20260927 00:42:20.298774 10990 leader_election.cc:302] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f8eb5cd0df4ed9a8cdd3b57a3508ad, 380815f2d0aa45758635942a23596889; no voters:
I20260927 00:42:20.298990 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.294757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58931 (local address 127.0.98.3:40555)
0927 00:42:20.294887 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.294893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.294908 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:42:20.295067 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.295070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.295116 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.295183 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.295590 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.296138 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.297693 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.298258 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.298351 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.298429 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.298792 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.298859 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:42:20.299075 11128 raft_consensus.cc:2781] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:42:20.299175 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.294456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39501 (local address 127.0.98.1:36319)
0927 00:42:20.295181 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.295195 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.295299 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.295624 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.295627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.295645 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.295786 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.295792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.296696 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.296701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.297877 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.297887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.298725 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.298728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.298829 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.298848 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.298948 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.299016 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260927 00:42:20.299178 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.294565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36319 (local address 127.0.98.2:39501)
0927 00:42:20.294668 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.294674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.295296 (+ 622us) server_negotiation.cc:407] Connection header received
0927 00:42:20.295399 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.295402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.295444 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.295519 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:42:20.295911 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.296562 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.298030 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.298610 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.298683 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.298738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.298987 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.299061 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:42:20.299278 11128 raft_consensus.cc:686] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 1 LEADER]: Becoming Leader. State: Replica: 19f8eb5cd0df4ed9a8cdd3b57a3508ad, State: Running, Role: LEADER
I20260927 00:42:20.299620 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.294395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40555 (local address 127.0.98.1:58931)
0927 00:42:20.294841 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.294854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:42:20.294957 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:42:20.295326 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:42:20.295332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:42:20.295345 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:42:20.295471 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.295476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.296254 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.296256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.297555 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:42:20.297564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.298526 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.298529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:42:20.298629 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.298660 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 00:42:20.299400 (+ 740us) client_negotiation.cc:240] Negotiation successful
0927 00:42:20.299481 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":27,"thread_start_us":117,"threads_started":1}
I20260927 00:42:20.299641 10923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "380815f2d0aa45758635942a23596889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c3b476860cd4ce09d3b715b167d76c0" is_pre_election: true
I20260927 00:42:20.299654 11128 consensus_queue.cc:237] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } }
I20260927 00:42:20.299810 10923 raft_consensus.cc:2379] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 380815f2d0aa45758635942a23596889 in current term 1: Already voted for candidate 19f8eb5cd0df4ed9a8cdd3b57a3508ad in this term.
I20260927 00:42:20.300257 11057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e713ee1bb540538db704edecbfd511" candidate_uuid: "380815f2d0aa45758635942a23596889" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" is_pre_election: true
I20260927 00:42:20.300817 10723 leader_election.cc:302] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380815f2d0aa45758635942a23596889; no voters: 19f8eb5cd0df4ed9a8cdd3b57a3508ad, 8c3b476860cd4ce09d3b715b167d76c0
I20260927 00:42:20.301069 11127 raft_consensus.cc:2726] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:42:20.302855 10644 catalog_manager.cc:5495] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 19f8eb5cd0df4ed9a8cdd3b57a3508ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f8eb5cd0df4ed9a8cdd3b57a3508ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 } health_report { overall_health: UNKNOWN } } }
I20260927 00:42:20.322024 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.318772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.3:40555)
0927 00:42:20.318896 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:42:20.318900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:42:20.318914 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:42:20.319067 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:42:20.319070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:42:20.319120 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:42:20.319213 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:42:20.319423 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.319900 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.320668 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:42:20.320865 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:42:20.321008 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:42:20.321277 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:42:20.321706 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:42:20.321806 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 00:42:20.321864 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:42:20.321906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:42:20.327643 10923 raft_consensus.cc:1261] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 1 FOLLOWER]: Refusing update from remote peer 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:20.327735 10789 raft_consensus.cc:1261] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 1 FOLLOWER]: Refusing update from remote peer 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:42:20.328343 11128 consensus_queue.cc:1035] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3b476860cd4ce09d3b715b167d76c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:20.328522 11132 consensus_queue.cc:1035] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "380815f2d0aa45758635942a23596889" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:42:20.334897 11135 mvcc.cc:205] Tried to move back new op lower bound from 7333764056375226368 to 7333764056270348288. Current Snapshot: MvccSnapshot[applied={T|T < 7333764056375226368}]
I20260927 00:42:20.335338 11137 mvcc.cc:205] Tried to move back new op lower bound from 7333764056375226368 to 7333764056270348288. Current Snapshot: MvccSnapshot[applied={T|T < 7333764056375226368}]
I20260927 00:42:20.336305 11136 mvcc.cc:205] Tried to move back new op lower bound from 7333764056375226368 to 7333764056270348288. Current Snapshot: MvccSnapshot[applied={T|T < 7333764056375226368}]
I20260927 00:42:20.343860 10642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:20.344028 10642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:42:20.346415 10642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 073cc1f4e56c492fb2754c5013f2fd48: Sending DeleteTablet for 3 replicas of tablet 08e713ee1bb540538db704edecbfd511
I20260927 00:42:20.347165 11037 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e713ee1bb540538db704edecbfd511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55026
I20260927 00:42:20.347239 10903 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e713ee1bb540538db704edecbfd511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45698
I20260927 00:42:20.347627 10769 tablet_service.cc:1514] Processing DeleteTablet for tablet 08e713ee1bb540538db704edecbfd511 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:42:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34152
I20260927 00:42:20.347674 11147 tablet_replica.cc:330] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: stopping tablet replica
I20260927 00:42:20.347975 11147 raft_consensus.cc:2229] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:42:20.347986 11148 tablet_replica.cc:330] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889: stopping tablet replica
I20260927 00:42:20.348248 11148 raft_consensus.cc:2229] T 08e713ee1bb540538db704edecbfd511 P 380815f2d0aa45758635942a23596889 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:20.348347 11147 raft_consensus.cc:2258] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:20.348695 11146 tablet_replica.cc:330] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: stopping tablet replica
I20260927 00:42:20.348740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10706
I20260927 00:42:20.349054 11146 raft_consensus.cc:2229] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:42:20.349340 11146 raft_consensus.cc:2258] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:42:20.350620 11147 ts_tablet_manager.cc:1905] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:20.350786 11146 ts_tablet_manager.cc:1905] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:42:20.353578 11146 ts_tablet_manager.cc:1918] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:20.353770 11146 log.cc:1196] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/08e713ee1bb540538db704edecbfd511
I20260927 00:42:20.354099 11147 ts_tablet_manager.cc:1918] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:42:20.354208 11147 log.cc:1196] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/08e713ee1bb540538db704edecbfd511
I20260927 00:42:20.354300 11146 ts_tablet_manager.cc:1939] T 08e713ee1bb540538db704edecbfd511 P 8c3b476860cd4ce09d3b715b167d76c0: Deleting consensus metadata
I20260927 00:42:20.354523 11147 ts_tablet_manager.cc:1939] T 08e713ee1bb540538db704edecbfd511 P 19f8eb5cd0df4ed9a8cdd3b57a3508ad: Deleting consensus metadata
I20260927 00:42:20.355628 10628 catalog_manager.cc:4841] TS 8c3b476860cd4ce09d3b715b167d76c0 (127.0.98.2:39501): tablet 08e713ee1bb540538db704edecbfd511 (table test-table [id=1b6e56f5fe3a474592305c247f1f007d]) successfully deleted
I20260927 00:42:20.355834 10629 catalog_manager.cc:4841] TS 19f8eb5cd0df4ed9a8cdd3b57a3508ad (127.0.98.3:40555): tablet 08e713ee1bb540538db704edecbfd511 (table test-table [id=1b6e56f5fe3a474592305c247f1f007d]) successfully deleted
W20260927 00:42:20.357867 10628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35261 (error 108)
I20260927 00:42:20.357995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10840
I20260927 00:42:20.359402 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:42:20.359074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35261 (local address 127.0.0.1:34164)
0927 00:42:20.359230 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 00:42:20.359303 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:42:20.365236 10628 catalog_manager.cc:4584] TS 380815f2d0aa45758635942a23596889 (127.0.98.1:35261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08e713ee1bb540538db704edecbfd511: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35261: connect: Connection refused (error 111)
I20260927 00:42:20.365382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10974
I20260927 00:42:20.369910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 10612
2026-09-27T00:42:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2279 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:42:20 dist-test-slave-pkws krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:42:20 dist-test-slave-pkws krb5kdc[11152](info): set up 2 sockets
Sep 27 00:42:20 dist-test-slave-pkws krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260927 00:42:21.409847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:42:21 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:42:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:42:21.574904 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskql8uh0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:42:24.058 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:42:24.058 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 41739
2026-09-27 00:42:24.061 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.41739"
2026-09-27 00:42:24.083 UTC [11185] LOG: database system was shut down at 2026-09-27 00:42:22 UTC
2026-09-27 00:42:24.088 UTC [11182] LOG: database system is ready to accept connections
I20260927 00:42:25.085731 392 mini_postgres.cc:96] Postgres bound to 41739
2026-09-27 00:42:25.092 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:41739 - accepting connections
I20260927 00:42:25.092583 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:42:25.106858 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:42:25.577612 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:42:25.577688 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin
2026-09-27 00:42:25,859 [I] DB FLAVOR :POSTGRES
2026-09-27 00:42:25,860 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:42:25,860 [I] Checking connection..
2026-09-27 00:42:25,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:42:26,118 [I] Checking connection passed.
2026-09-27 00:42:26,119 [I] --------- Verifying version history table ---------
2026-09-27 00:42:26,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:42:26,350 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:42:26,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:42:26,583 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:42:26,584 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:42:26,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:42:26.835 UTC [11257] WARNING: there is no transaction in progress
2026-09-27 00:42:26,841 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:42:26,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:42:27,078 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:42:27,079 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:42:27,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:42:27,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:42:27,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pkws.c.gcp-upstream.internal','N') ;"
2026-09-27 00:42:27,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:42:28,061 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:42:28,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:42:28,285 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:42:28,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:42:28,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:42:28,815 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:42:28,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:42:29.715 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.731 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.746 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.760 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.778 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.797 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.860 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.866 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.880 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.889 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.899 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.912 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.921 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.926 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.932 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:29.938 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 00:42:30,571 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:42:30,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:42:30,788 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:42:30,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:42:31,046 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:42:31,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:42:31,270 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:42:31,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:42:31,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:42:31,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pkws.c.gcp-upstream.internal';"
2026-09-27 00:42:31,989 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:42:31,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:42:32,209 [I] DB_PATCHES have already been applied
I20260927 00:42:32.215276 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskql8uh0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:42:32.215344 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:42:32.218740 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52249
Sep 27, 2026 12:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:42:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:42:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:42:33.310797 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.218s user 0.004s sys 0.005s
Sep 27, 2026 12:42:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:42:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 12:42:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:42:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:42:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:42:53.705646 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:42:53 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:42:53.806205 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:42:53.816229 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:42:54.683748 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:42:54.683840 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms
I20260927 00:42:54.683940 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41739
2026-09-27 00:42:54,868 [I] DB FLAVOR :POSTGRES
2026-09-27 00:42:54,868 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:42:54,868 [I] Checking connection
2026-09-27 00:42:54,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:42:55,109 [I] connection success
2026-09-27 00:42:55,109 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:42:55,109 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:42:55,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:42:55,358 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:42:55,358 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:42:55,359 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:42:55,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:42:55,605 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:42:56.066222 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:42:56.257273 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:42:56.299515 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:42:56 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469773, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:42:56.690969 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:42:56.691131 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:42:56.691152 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:42:56.695915 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260927 00:42:56.696054 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52471
Sep 27, 2026 12:42:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:42:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:42:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:42:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:42:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 00:42:58.987221 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.181s user 0.010s sys 0.159s
I20260927 00:42:58.987370 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:42:58.987502 392 mini_ranger_kms.cc:401] 127.0.98.20:52471/kms/v1/keys
Sep 27 00:42:59 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 00:42:59 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:42:59 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469779, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:43:02 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:02.254034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37373
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52471/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:02.366364 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:02.366622 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:02.366660 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:02.370606 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:02.370661 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:02.370680 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:02.370697 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:02.370712 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:02.376016 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52471/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37373
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:02.376973 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:02.377918 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:02.386890 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:02.386890 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:02.386968 11596 server_base.cc:1034] running on GCE node
W20260927 00:43:02.386929 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:02.387735 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:02.388808 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:02.389999 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790469782389969 us; error 40 us; skew 500 ppm
Sep 27 00:43:02 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:02.395778 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:02.396651 11596 webserver.cc:460] Webserver started at http://127.0.98.62:41281/ using document root <none> and password file <none>
I20260927 00:43:02.397212 11596 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:02.397269 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:02.397431 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:02 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:02.442930 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "32d164fc007e440c814e6bc368fdb561"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-pkws"
server_key: "90721c932cf92d39cff26bb9a761b514"
server_key_iv: "f821935f46afae337c81f72bc1dde666"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:02.443624 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "32d164fc007e440c814e6bc368fdb561"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-pkws"
server_key: "90721c932cf92d39cff26bb9a761b514"
server_key_iv: "f821935f46afae337c81f72bc1dde666"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:02.447629 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:02.462037 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:02.463438 11596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:02.463577 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "32d164fc007e440c814e6bc368fdb561"
format_stamp: "Formatted at 2026-09-27 00:43:02 on dist-test-slave-pkws"
server_key: "90721c932cf92d39cff26bb9a761b514"
server_key_iv: "f821935f46afae337c81f72bc1dde666"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:02.463721 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:02.483768 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:02.486788 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:02.487007 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:02.495245 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37373
I20260927 00:43:02.495262 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37373 every 8 connection(s)
I20260927 00:43:02.496245 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:02.498994 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:02.500860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 11596
I20260927 00:43:02.501014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:02.501320 392 external_mini_cluster.cc:1428] Setting key ba5836b906d30713e5d841938d4b9f3e
I20260927 00:43:02.505899 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: Bootstrap starting.
I20260927 00:43:02.508744 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:02.509541 11664 log.cc:824] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:43:02 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:02.511787 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: No bootstrap required, opened a new log
I20260927 00:43:02.515735 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } }
I20260927 00:43:02.516038 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:02.516292 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d164fc007e440c814e6bc368fdb561, State: Initialized, Role: FOLLOWER
I20260927 00:43:02.516826 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } }
I20260927 00:43:02.517016 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:02.517122 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:02.517227 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:02.517280 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:02.502743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.0.98.62:37373)
0927 00:43:02.503227 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:02.503247 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:02.503293 (+ 46us) server_negotiation.cc:407] Connection header received
0927 00:43:02.504187 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:02.504211 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:02.504617 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:02.505017 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:02.505848 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.507399 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.508557 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:02.509308 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:02.509462 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:02.512081 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:02.512087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:02.512108 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:02.512133 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:02.514312 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:02.514759 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:02.514770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:02.514774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:02.514880 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:02.515169 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:02.515172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:02.515173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:02.515536 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:02.515752 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:02.516065 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 00:43:02.516337 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":131,"threads_started":1}
I20260927 00:43:02.518298 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } }
I20260927 00:43:02.518656 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32d164fc007e440c814e6bc368fdb561; no voters:
I20260927 00:43:02.518982 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:02.519076 11669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:02.519618 11669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [term 1 LEADER]: Becoming Leader. State: Replica: 32d164fc007e440c814e6bc368fdb561, State: Running, Role: LEADER
I20260927 00:43:02.519987 11669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } }
I20260927 00:43:02.520592 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:02.521818 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32d164fc007e440c814e6bc368fdb561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } } }
I20260927 00:43:02.521947 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:02.522212 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d164fc007e440c814e6bc368fdb561. Latest consensus state: current_term: 1 leader_uuid: "32d164fc007e440c814e6bc368fdb561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32d164fc007e440c814e6bc368fdb561" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37373 } } }
I20260927 00:43:02.522449 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:02.522332 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:02.522845 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:43:02.525449 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:02.539710 11678 catalog_manager.cc:1347] Generated new cluster ID: 36a31bcb7c3640dda8918e601b4b03ee
I20260927 00:43:02.539783 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:02.558727 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:02.560288 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:02.569634 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: Generated new TSK 0
I20260927 00:43:02.570585 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:43:03.274307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52471/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:03.431507 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:03.431950 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:03.432075 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:03.438179 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:03.438254 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:03.438359 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:03.446302 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52471/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:03.447647 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:03.449007 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:03.461853 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:03.461895 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:03.462368 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:03.462458 11709 server_base.cc:1034] running on GCE node
I20260927 00:43:03.463068 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:03.463835 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:03.465018 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790469783464977 us; error 56 us; skew 500 ppm
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:03.468657 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:03.469758 11709 webserver.cc:460] Webserver started at http://127.0.98.1:40035/ using document root <none> and password file <none>
I20260927 00:43:03.470444 11709 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:03.470530 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:03.470744 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:03.487242 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b4dbc1da041472c914d0ac2d2b42ad3"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "9b901d008678e7398de7086e1b449e7d"
server_key_iv: "4d33772cc435fafdeee97f5a37594995"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.488045 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b4dbc1da041472c914d0ac2d2b42ad3"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "9b901d008678e7398de7086e1b449e7d"
server_key_iv: "4d33772cc435fafdeee97f5a37594995"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.498368 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260927 00:43:03.537552 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:03.539304 11709 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:03.539448 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "0b4dbc1da041472c914d0ac2d2b42ad3"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "9b901d008678e7398de7086e1b449e7d"
server_key_iv: "4d33772cc435fafdeee97f5a37594995"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.539608 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:03.558229 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:03.562251 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:03.562506 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:03.563246 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:03.564301 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:03.564414 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:03.564540 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:03.564620 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:03.578542 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35343
I20260927 00:43:03.579854 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:03.580865 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35343 every 8 connection(s)
I20260927 00:43:03.583232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 11709
I20260927 00:43:03.583364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:03.583680 392 external_mini_cluster.cc:1428] Setting key b1ba372aac52cd13a7cd2244316eb457
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:03.608132 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.584824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37373 (local address 127.0.98.1:41129)
0927 00:43:03.589265 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:03.589321 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:03.590345 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:03.592284 (+ 1939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:03.592294 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:03.592682 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:03.593417 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:03.593437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.594708 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.594712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:03.596176 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:03.596188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.596982 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.596988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:03.597206 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:03.597898 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:03.597919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:03.602492 (+ 4573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:03.604816 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:03.604822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:03.604838 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:03.605214 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:03.605522 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:03.605525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:03.605527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:03.605712 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:03.606201 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:03.606209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:03.606563 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 00:43:03.606767 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 00:43:03.607102 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4196,"thread_start_us":175,"threads_started":1}
I20260927 00:43:03.608861 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.585963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41129 (local address 127.0.98.62:37373)
0927 00:43:03.591806 (+ 5843us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:03.591811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:03.591827 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:43:03.591869 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:03.591872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:03.591924 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:03.593579 (+ 1655us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:03.593696 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.594548 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.596323 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.596689 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.596788 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:03.602708 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:03.602713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:03.602716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:03.602744 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:03.604678 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:03.605334 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:03.605337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:03.605339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:03.605424 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:03.605826 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:03.605829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:03.605831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:03.606053 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:03.606149 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:03.608566 (+ 2417us) server_negotiation.cc:299] Negotiation successful
0927 00:43:03.608692 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5761,"thread_start_us":63,"threads_started":1}
I20260927 00:43:03.610620 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37373
I20260927 00:43:03.610855 11842 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:03.611467 11842 heartbeater.cc:511] Master 127.0.98.62:37373 requested a full tablet report, sending...
I20260927 00:43:03.613364 11628 ts_manager.cc:194] Registered new tserver with Master: 0b4dbc1da041472c914d0ac2d2b42ad3 (127.0.98.1:35343)
I20260927 00:43:03.615044 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:43:03.620759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52471/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:43:03.777241 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:03.777680 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:03.777805 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:03.783809 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:03.783871 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:03.783964 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:03.790366 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52471/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:03.791527 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:03.792541 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:03.803476 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:03.803510 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:03.803579 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:03.805141 11848 server_base.cc:1034] running on GCE node
I20260927 00:43:03.805615 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:03.806427 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:03.807636 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790469783807616 us; error 41 us; skew 500 ppm
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:03.811376 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:03.812388 11848 webserver.cc:460] Webserver started at http://127.0.98.2:33943/ using document root <none> and password file <none>
I20260927 00:43:03.813023 11848 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:03.813088 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:03.813306 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:03.828365 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "61183a7866d64a32997dae5aa08dde3d"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "0609e93218ed5676a8de98744ddd0af5"
server_key_iv: "73669abff3f918d1e20e70bb93a6529a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.829133 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61183a7866d64a32997dae5aa08dde3d"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "0609e93218ed5676a8de98744ddd0af5"
server_key_iv: "73669abff3f918d1e20e70bb93a6529a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.834556 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 00:43:03.862115 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:03.864280 11848 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260927 00:43:03.864418 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "61183a7866d64a32997dae5aa08dde3d"
format_stamp: "Formatted at 2026-09-27 00:43:03 on dist-test-slave-pkws"
server_key: "0609e93218ed5676a8de98744ddd0af5"
server_key_iv: "73669abff3f918d1e20e70bb93a6529a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:03.864548 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:03.886869 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:03.891793 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:03.892027 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:03.893296 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:03.894534 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:03.894608 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:03.894675 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:03.894759 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 00:43:03.911479 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43795
I20260927 00:43:03.911501 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43795 every 8 connection(s)
I20260927 00:43:03.912707 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:03.917438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 11848
I20260927 00:43:03.917559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:03.917876 392 external_mini_cluster.cc:1428] Setting key 2c23c31832c77c5c82f4b25e67f720df
Sep 27 00:43:03 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:43:03.931636 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.917782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37669 (local address 127.0.98.62:37373)
0927 00:43:03.917917 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:03.917921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:03.918496 (+ 575us) server_negotiation.cc:407] Connection header received
0927 00:43:03.919764 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:03.919768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:03.919820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:03.919873 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:03.921524 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.922362 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.924072 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.924431 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.924503 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:03.927468 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:03.927472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:03.927475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:03.927506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:03.929248 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:03.929848 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:03.929852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:03.929854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:03.929940 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:03.930320 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:03.930323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:03.930325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:03.930527 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:03.930610 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:03.931329 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 00:43:03.931465 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:03.933483 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:03.916608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37373 (local address 127.0.98.2:37669)
0927 00:43:03.918335 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:03.918395 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:03.919481 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:03.920286 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:03.920296 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:03.920658 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:03.921367 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:03.921387 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.922513 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.922516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:03.923931 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:03.923942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:03.924692 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:03.924695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:03.924929 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:03.925592 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:03.925611 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:03.927277 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:03.929379 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:03.929384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:03.929400 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:03.929718 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:03.930047 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:03.930050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:03.930052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:03.930215 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:03.930678 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:03.930684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:03.931024 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 00:43:03.932095 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0927 00:43:03.932428 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1489,"thread_start_us":174,"threads_started":1}
I20260927 00:43:03.935134 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37373
I20260927 00:43:03.935403 11978 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:43:03.936007 11978 heartbeater.cc:511] Master 127.0.98.62:37373 requested a full tablet report, sending...
I20260927 00:43:03.938256 11628 ts_manager.cc:194] Registered new tserver with Master: 61183a7866d64a32997dae5aa08dde3d (127.0.98.2:43795)
I20260927 00:43:03.939010 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:43:03.955983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52471/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:04 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:43:04.117748 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:04.118171 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:04.118269 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:04.124202 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:04.124392 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:04.124522 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:04.132098 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52471/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:04.133673 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:04.135036 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:04.147461 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.147480 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:04.147466 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:04.148314 11983 server_base.cc:1034] running on GCE node
I20260927 00:43:04.148875 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:04.149673 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:04.152971 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790469784152899 us; error 82 us; skew 500 ppm
Sep 27 00:43:04 dist-test-slave-pkws krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:04.156816 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:04.157892 11983 webserver.cc:460] Webserver started at http://127.0.98.3:38325/ using document root <none> and password file <none>
I20260927 00:43:04.158628 11983 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:04.158757 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:04.159026 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:04 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:04.180325 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "b47e25dd8e034674a3570fb6694c7133"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-pkws"
server_key: "c99615f20b31e92790e408e63ba4b9d0"
server_key_iv: "d555e17caf2a77d0afbde215e9b14cd9"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:04.181355 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b47e25dd8e034674a3570fb6694c7133"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-pkws"
server_key: "c99615f20b31e92790e408e63ba4b9d0"
server_key_iv: "d555e17caf2a77d0afbde215e9b14cd9"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:04.200801 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.000s sys 0.004s
I20260927 00:43:04.221205 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.222970 11983 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 00:43:04.223109 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b47e25dd8e034674a3570fb6694c7133"
format_stamp: "Formatted at 2026-09-27 00:43:04 on dist-test-slave-pkws"
server_key: "c99615f20b31e92790e408e63ba4b9d0"
server_key_iv: "d555e17caf2a77d0afbde215e9b14cd9"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:04.223245 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:04.245898 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:04.249444 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:04.249630 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:04.250267 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:04.251199 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:04.251257 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.251314 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:04.251344 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:04.261464 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42043
I20260927 00:43:04.262559 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:04.268015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 11983
I20260927 00:43:04.268138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:04.268435 392 external_mini_cluster.cc:1428] Setting key e3bc3fd8211bc30dbace22cc118e93fa
I20260927 00:43:04.269313 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42043 every 8 connection(s)
Sep 27 00:43:04 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:04.280395 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.264845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53349 (local address 127.0.98.62:37373)
0927 00:43:04.264989 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.264993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.266505 (+ 1512us) server_negotiation.cc:407] Connection header received
0927 00:43:04.267620 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.267626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.267686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.267800 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:04.270449 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.271278 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.272999 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.273358 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.273484 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.276303 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:04.276307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:04.276310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:04.276336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:04.278035 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.278640 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.278644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.278646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.278728 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.279110 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.279113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.279115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.279330 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:04.279423 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.280102 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.280225 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 00:43:04.281369 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.265196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37373 (local address 127.0.98.3:53349)
0927 00:43:04.266331 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.266382 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.267376 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.269012 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.269025 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.269370 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:04.270261 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.270277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.271503 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.271506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.272873 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.272883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.273521 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.273525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.273720 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.274406 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:04.274426 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:04.276097 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:04.278162 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.278168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.278182 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.278524 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.278825 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:04.278828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:04.278830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:04.279003 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:04.279450 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:04.279456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:04.279818 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.280029 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.280337 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":126,"threads_started":1}
I20260927 00:43:04.283074 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37373
I20260927 00:43:04.283311 12112 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:04.283823 12112 heartbeater.cc:511] Master 127.0.98.62:37373 requested a full tablet report, sending...
I20260927 00:43:04.284926 11628 ts_manager.cc:194] Registered new tserver with Master: b47e25dd8e034674a3570fb6694c7133 (127.0.98.3:42043)
I20260927 00:43:04.285640 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53349
I20260927 00:43:04.295310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:04.308770 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.297537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.98.62:37373)
0927 00:43:04.297646 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.297651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.298183 (+ 532us) server_negotiation.cc:407] Connection header received
0927 00:43:04.298302 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.298305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.298352 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.298429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:04.299421 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.300232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.302135 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.302472 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.302542 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.304328 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:04.304333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:04.304337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:04.304364 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:04.306164 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.307489 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.307492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.307494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.307587 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:04.307898 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:04.307901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:04.307903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:04.308106 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:04.308233 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.308478 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.308597 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:43:04.312389 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:04.317083 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 27 00:43:04 dist-test-slave-pkws krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:04.344014 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.329407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43795 (local address 127.0.0.1:53602)
0927 00:43:04.332361 (+ 2954us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.332372 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.332441 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.337914 (+ 5473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.337917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.337935 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.338029 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.338037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.341111 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.341115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.342215 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.342225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.343512 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.343516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.343610 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.343663 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.343746 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.343860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2902,"thread_start_us":77,"threads_started":1}
I20260927 00:43:04.345064 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.329693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.2:43795)
0927 00:43:04.337472 (+ 7779us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.337483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.337504 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:43:04.337573 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.337578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.337763 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.338193 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.338353 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.339396 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.342451 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.343382 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.344585 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.344741 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.344793 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.344901 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7667,"thread_start_us":108,"threads_started":1}
I20260927 00:43:04.345224 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.329088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35343 (local address 127.0.0.1:51126)
0927 00:43:04.332102 (+ 3014us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.332115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.332188 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.339539 (+ 7351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.339542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.339579 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.339655 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.339660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.342406 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.342410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.343555 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.343565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.344749 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.344753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.344849 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.344873 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.344945 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.345010 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2954,"thread_start_us":97,"threads_started":1}
I20260927 00:43:04.345850 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.329237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.0.98.1:35343)
0927 00:43:04.336815 (+ 7578us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.336825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.336853 (+ 28us) server_negotiation.cc:407] Connection header received
0927 00:43:04.336952 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.336957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.337168 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.339802 (+ 2634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.339938 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.340982 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.343712 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.344577 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.345402 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.345548 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.345616 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.345715 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7456,"thread_start_us":78,"threads_started":1}
I20260927 00:43:04.348651 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 (DEFAULT_TABLE table=test-table [id=8f5fd7edf7fe4d2e9c94f5cb81ac1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:04.348668 11904 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 (DEFAULT_TABLE table=test-table [id=8f5fd7edf7fe4d2e9c94f5cb81ac1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:04.349766 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa0630fd3e947449c599d0c54ee16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:04.349768 11904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa0630fd3e947449c599d0c54ee16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:04.354035 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.327893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42043 (local address 127.0.0.1:48866)
0927 00:43:04.330579 (+ 2686us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.330628 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.330752 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.331740 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.331746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.331769 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.331896 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.331903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.334763 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.334767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.336509 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.336521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.337987 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.337994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.338132 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.353744 (+ 15612us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.353814 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.353892 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2503,"thread_start_us":106,"threads_started":1}
I20260927 00:43:04.358407 12131 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Bootstrap starting.
I20260927 00:43:04.359527 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.328814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.3:42043)
0927 00:43:04.331034 (+ 2220us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.331048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.331077 (+ 29us) server_negotiation.cc:407] Connection header received
0927 00:43:04.331185 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.331191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.331419 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.331570 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.332960 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.334625 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.336677 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.337591 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.356702 (+ 19111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.359272 (+ 2570us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.359328 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.359415 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2111,"thread_start_us":959,"threads_started":1}
I20260927 00:43:04.360692 12131 tablet_bootstrap.cc:654] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:04.362062 12131 log.cc:824] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:04.362464 12132 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Bootstrap starting.
I20260927 00:43:04.362680 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 (DEFAULT_TABLE table=test-table [id=8f5fd7edf7fe4d2e9c94f5cb81ac1ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:04.363765 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa0630fd3e947449c599d0c54ee16b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:04.364537 12132 tablet_bootstrap.cc:654] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:04.365813 12132 log.cc:824] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:04.368152 12131 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: No bootstrap required, opened a new log
I20260927 00:43:04.368398 12131 ts_tablet_manager.cc:1397] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260927 00:43:04.372792 12131 raft_consensus.cc:348] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.373206 12131 raft_consensus.cc:374] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:04.373327 12131 raft_consensus.cc:729] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4dbc1da041472c914d0ac2d2b42ad3, State: Initialized, Role: FOLLOWER
I20260927 00:43:04.373898 12135 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Bootstrap starting.
I20260927 00:43:04.374971 12132 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: No bootstrap required, opened a new log
I20260927 00:43:04.375233 12132 ts_tablet_manager.cc:1397] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260927 00:43:04.377885 12135 tablet_bootstrap.cc:654] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:04.378350 12131 consensus_queue.cc:260] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.378773 12135 log.cc:824] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:04.380340 12132 raft_consensus.cc:348] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.380602 12132 raft_consensus.cc:374] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:04.380645 12132 raft_consensus.cc:729] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61183a7866d64a32997dae5aa08dde3d, State: Initialized, Role: FOLLOWER
I20260927 00:43:04.381448 12131 ts_tablet_manager.cc:1428] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 00:43:04.381866 11842 heartbeater.cc:503] Master 127.0.98.62:37373 was elected leader, sending a full tablet report...
I20260927 00:43:04.386169 12132 consensus_queue.cc:260] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.387902 12132 ts_tablet_manager.cc:1428] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 00:43:04.388322 11978 heartbeater.cc:503] Master 127.0.98.62:37373 was elected leader, sending a full tablet report...
I20260927 00:43:04.393636 12135 tablet_bootstrap.cc:492] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: No bootstrap required, opened a new log
I20260927 00:43:04.393895 12135 ts_tablet_manager.cc:1397] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260927 00:43:04.398025 12135 raft_consensus.cc:348] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.398406 12135 raft_consensus.cc:374] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:04.398504 12135 raft_consensus.cc:729] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47e25dd8e034674a3570fb6694c7133, State: Initialized, Role: FOLLOWER
I20260927 00:43:04.399816 12135 consensus_queue.cc:260] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.402117 12135 ts_tablet_manager.cc:1428] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 00:43:04.402734 12112 heartbeater.cc:503] Master 127.0.98.62:37373 was elected leader, sending a full tablet report...
W20260927 00:43:04.414462 11979 tablet.cc:2369] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:04.520226 12113 tablet.cc:2369] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:04.582720 11843 tablet.cc:2369] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:04.613166 12139 raft_consensus.cc:482] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:04.613422 12139 raft_consensus.cc:504] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.614869 12139 leader_election.cc:288] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b4dbc1da041472c914d0ac2d2b42ad3 (127.0.98.1:35343), 61183a7866d64a32997dae5aa08dde3d (127.0.98.2:43795)
I20260927 00:43:04.617182 12138 raft_consensus.cc:482] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:04.617372 12138 raft_consensus.cc:504] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.618634 12138 leader_election.cc:288] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b4dbc1da041472c914d0ac2d2b42ad3 (127.0.98.1:35343), b47e25dd8e034674a3570fb6694c7133 (127.0.98.3:42043)
I20260927 00:43:04.622320 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.615302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35343 (local address 127.0.98.3:52631)
0927 00:43:04.615995 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.616010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.616114 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.617138 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.617141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.617161 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.617288 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.617295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.618572 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.618576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.619861 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.619879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.621874 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.621877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.621979 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.622005 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.622102 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.622178 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612}
I20260927 00:43:04.623030 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.616386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52631 (local address 127.0.98.1:35343)
0927 00:43:04.616687 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.616691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.616707 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:43:04.616762 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.616765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.616820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.617502 (+ 682us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.617621 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.618352 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.621062 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.621645 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.621751 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.621815 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.622834 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.622912 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260927 00:43:04.623661 11781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "b47e25dd8e034674a3570fb6694c7133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" is_pre_election: true
I20260927 00:43:04.623955 11781 raft_consensus.cc:2454] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b47e25dd8e034674a3570fb6694c7133 in term 0.
I20260927 00:43:04.624933 12000 leader_election.cc:302] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4dbc1da041472c914d0ac2d2b42ad3, b47e25dd8e034674a3570fb6694c7133; no voters:
I20260927 00:43:04.625296 12139 raft_consensus.cc:2781] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:04.625381 12139 raft_consensus.cc:482] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:04.625424 12139 raft_consensus.cc:3037] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:04.627264 12139 raft_consensus.cc:504] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.627641 12139 leader_election.cc:288] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [CANDIDATE]: Term 1 election: Requested vote from peers 0b4dbc1da041472c914d0ac2d2b42ad3 (127.0.98.1:35343), 61183a7866d64a32997dae5aa08dde3d (127.0.98.2:43795)
I20260927 00:43:04.628326 11781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "b47e25dd8e034674a3570fb6694c7133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3"
I20260927 00:43:04.628453 11781 raft_consensus.cc:3037] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:04.629828 11781 raft_consensus.cc:2454] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47e25dd8e034674a3570fb6694c7133 in term 1.
I20260927 00:43:04.633682 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.620480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42043 (local address 127.0.98.2:52413)
0927 00:43:04.624001 (+ 3521us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.624015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.624115 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.629867 (+ 5752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.629871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.629888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.630000 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.630007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.631356 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.631359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.632233 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.632242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.633223 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.633226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.633348 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.633371 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.633440 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.633515 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3455,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260927 00:43:04.633692 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.618943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35343 (local address 127.0.98.2:46783)
0927 00:43:04.623569 (+ 4626us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.623585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.623689 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.629462 (+ 5773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.629466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.629483 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.629594 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.629601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.630597 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.630602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.632416 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.632423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.633313 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.633320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.633388 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.633422 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.633482 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.633528 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4559,"mutex_wait_us":732,"thread_start_us":91,"threads_started":1}
I20260927 00:43:04.633961 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.620106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46783 (local address 127.0.98.1:35343)
0927 00:43:04.623182 (+ 3076us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.623186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.629252 (+ 6066us) server_negotiation.cc:407] Connection header received
0927 00:43:04.629311 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.629313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.629348 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.629446 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.629724 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.630455 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.632525 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.633128 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.633217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.633265 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.633741 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.633822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3020,"thread_start_us":112,"threads_started":1}
I20260927 00:43:04.634262 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.620817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52413 (local address 127.0.98.3:42043)
0927 00:43:04.620959 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.620964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.624189 (+ 3225us) server_negotiation.cc:407] Connection header received
0927 00:43:04.624236 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.624239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.624280 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.624353 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.630482 (+ 6129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.631229 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.632371 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.632989 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.633086 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.633147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.634063 (+ 916us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.634140 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:43:04.634393 11781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "61183a7866d64a32997dae5aa08dde3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" is_pre_election: true
I20260927 00:43:04.634575 11781 raft_consensus.cc:2379] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61183a7866d64a32997dae5aa08dde3d in current term 1: Already voted for candidate b47e25dd8e034674a3570fb6694c7133 in this term.
I20260927 00:43:04.634690 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "61183a7866d64a32997dae5aa08dde3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b47e25dd8e034674a3570fb6694c7133" is_pre_election: true
I20260927 00:43:04.634932 12066 raft_consensus.cc:2379] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61183a7866d64a32997dae5aa08dde3d in current term 1: Already voted for candidate b47e25dd8e034674a3570fb6694c7133 in this term.
I20260927 00:43:04.634999 12000 leader_election.cc:302] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4dbc1da041472c914d0ac2d2b42ad3, b47e25dd8e034674a3570fb6694c7133; no voters:
I20260927 00:43:04.635623 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.615573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43795 (local address 127.0.98.3:49409)
0927 00:43:04.629604 (+ 14031us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.629619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:04.629707 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:04.630269 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:04.630272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:04.630305 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:04.630376 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.630381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.632336 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.632340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.633505 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:04.633515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.635218 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.635222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:04.635315 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.635339 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:43:04.635414 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:43:04.635473 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13958,"thread_start_us":119,"threads_started":1}
I20260927 00:43:04.635850 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.616530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49409 (local address 127.0.98.2:43795)
0927 00:43:04.617002 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.617006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.629982 (+ 12976us) server_negotiation.cc:407] Connection header received
0927 00:43:04.630034 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.630037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.630082 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.630175 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:04.631376 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.632197 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.634200 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.634748 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.634846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.634898 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.635630 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.635737 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":389}
I20260927 00:43:04.636096 11865 leader_election.cc:302] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61183a7866d64a32997dae5aa08dde3d; no voters: 0b4dbc1da041472c914d0ac2d2b42ad3, b47e25dd8e034674a3570fb6694c7133
I20260927 00:43:04.636382 12138 raft_consensus.cc:3037] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:04.636938 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "b47e25dd8e034674a3570fb6694c7133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61183a7866d64a32997dae5aa08dde3d"
I20260927 00:43:04.636996 11920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa0630fd3e947449c599d0c54ee16b2" candidate_uuid: "b47e25dd8e034674a3570fb6694c7133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61183a7866d64a32997dae5aa08dde3d" is_pre_election: true
I20260927 00:43:04.637356 12139 raft_consensus.cc:2781] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:04.637614 12138 raft_consensus.cc:2726] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 00:43:04.637730 12139 raft_consensus.cc:686] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 LEADER]: Becoming Leader. State: Replica: b47e25dd8e034674a3570fb6694c7133, State: Running, Role: LEADER
I20260927 00:43:04.638067 12139 consensus_queue.cc:237] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } }
I20260927 00:43:04.638718 11932 raft_consensus.cc:2454] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47e25dd8e034674a3570fb6694c7133 in term 1.
I20260927 00:43:04.641251 11628 catalog_manager.cc:5495] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 reported cstate change: term changed from 0 to 1, leader changed from <none> to b47e25dd8e034674a3570fb6694c7133 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b47e25dd8e034674a3570fb6694c7133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b47e25dd8e034674a3570fb6694c7133" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42043 } health_report { overall_health: HEALTHY } } }
I20260927 00:43:04.702771 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.699202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.0.98.3:42043)
0927 00:43:04.699345 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:04.699351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:04.699366 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:43:04.699471 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:04.699473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:04.699523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:04.699665 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:04.699931 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.700433 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.701344 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:04.701555 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:04.701658 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:04.701962 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:04.702385 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:04.702518 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:04.702553 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 00:43:04.702606 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:43:04.708354 11932 raft_consensus.cc:1261] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 1 FOLLOWER]: Refusing update from remote peer b47e25dd8e034674a3570fb6694c7133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:04.708427 11781 raft_consensus.cc:1261] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 1 FOLLOWER]: Refusing update from remote peer b47e25dd8e034674a3570fb6694c7133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:04.709061 12139 consensus_queue.cc:1035] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61183a7866d64a32997dae5aa08dde3d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:04.709282 12152 consensus_queue.cc:1035] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4dbc1da041472c914d0ac2d2b42ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:04.715683 12155 mvcc.cc:205] Tried to move back new op lower bound from 7333764238158680064 to 7333764237880356864. Current Snapshot: MvccSnapshot[applied={T|T < 7333764238158680064}]
I20260927 00:43:04.717599 12156 mvcc.cc:205] Tried to move back new op lower bound from 7333764238158680064 to 7333764237880356864. Current Snapshot: MvccSnapshot[applied={T|T < 7333764238158680064}]
I20260927 00:43:04.724609 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:04.724776 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:04.727119 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32d164fc007e440c814e6bc368fdb561: Sending DeleteTablet for 3 replicas of tablet 3fa0630fd3e947449c599d0c54ee16b2
I20260927 00:43:04.727943 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51126
I20260927 00:43:04.727967 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48866
I20260927 00:43:04.728404 12164 tablet_replica.cc:330] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3: stopping tablet replica
I20260927 00:43:04.728562 11904 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa0630fd3e947449c599d0c54ee16b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53602
I20260927 00:43:04.728804 12164 raft_consensus.cc:2229] T 3fa0630fd3e947449c599d0c54ee16b2 P 0b4dbc1da041472c914d0ac2d2b42ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:04.729087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 11709
I20260927 00:43:04.729763 12166 tablet_replica.cc:330] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: stopping tablet replica
I20260927 00:43:04.730021 12166 raft_consensus.cc:2229] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:04.730361 12166 raft_consensus.cc:2258] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:04.731786 12166 ts_tablet_manager.cc:1905] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:04.733079 12165 tablet_replica.cc:330] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: stopping tablet replica
I20260927 00:43:04.733418 12165 raft_consensus.cc:2229] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:04.733860 12165 raft_consensus.cc:2258] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:04.735251 12165 ts_tablet_manager.cc:1905] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:43:04.735761 11613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35343 (error 108)
I20260927 00:43:04.736063 12166 ts_tablet_manager.cc:1918] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:04.736164 12166 log.cc:1196] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/3fa0630fd3e947449c599d0c54ee16b2
I20260927 00:43:04.736470 12166 ts_tablet_manager.cc:1939] T 3fa0630fd3e947449c599d0c54ee16b2 P 61183a7866d64a32997dae5aa08dde3d: Deleting consensus metadata
I20260927 00:43:04.736752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 11848
I20260927 00:43:04.738672 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.738358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35343 (local address 127.0.0.1:51132)
0927 00:43:04.738492 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.738569 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:43:04.738812 11613 catalog_manager.cc:4584] TS 0b4dbc1da041472c914d0ac2d2b42ad3 (127.0.98.1:35343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fa0630fd3e947449c599d0c54ee16b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35343: connect: Connection refused (error 111)
I20260927 00:43:04.739284 12165 ts_tablet_manager.cc:1918] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:04.739375 12165 log.cc:1196] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/3fa0630fd3e947449c599d0c54ee16b2
I20260927 00:43:04.739735 12165 ts_tablet_manager.cc:1939] T 3fa0630fd3e947449c599d0c54ee16b2 P b47e25dd8e034674a3570fb6694c7133: Deleting consensus metadata
I20260927 00:43:04.740869 11612 catalog_manager.cc:4841] TS b47e25dd8e034674a3570fb6694c7133 (127.0.98.3:42043): tablet 3fa0630fd3e947449c599d0c54ee16b2 (table test-table [id=8f5fd7edf7fe4d2e9c94f5cb81ac1ca9]) successfully deleted
I20260927 00:43:04.743208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 11983
I20260927 00:43:04.744546 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:04.744272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43795 (local address 127.0.0.1:53610)
0927 00:43:04.744407 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:04.744466 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:43:04.748546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 11596
I20260927 00:43:04.756139 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:43:05.090389 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:43:05.090504 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:43:05.436056 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:43:05.436 UTC [11182] LOG: received smart shutdown request
2026-09-27 00:43:05.439 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-09-27 00:43:05.439 UTC [11186] LOG: shutting down
2026-09-27 00:43:05.609 UTC [11182] LOG: database system is shut down
2026-09-27T00:43:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45358 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:05 dist-test-slave-pkws krb5kdc[12176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:05 dist-test-slave-pkws krb5kdc[12176](info): set up 2 sockets
Sep 27 00:43:05 dist-test-slave-pkws krb5kdc[12176](info): commencing operation
krb5kdc: starting...
W20260927 00:43:06.773084 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:06 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 00:43:06.933494 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskql8uh0/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/postgres -l logfile start
2026-09-27 00:43:09.380 UTC [12206] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 00:43:09.380 UTC [12206] LOG: listening on IPv4 address "127.0.98.20", port 45027
2026-09-27 00:43:09.384 UTC [12206] LOG: listening on Unix socket "/tmp/.s.PGSQL.45027"
2026-09-27 00:43:09.405 UTC [12209] LOG: database system was shut down at 2026-09-27 00:43:08 UTC
2026-09-27 00:43:09.408 UTC [12206] LOG: database system is ready to accept connections
I20260927 00:43:10.364754 392 mini_postgres.cc:96] Postgres bound to 45027
2026-09-27 00:43:10.370 UTC [12217] FATAL: database "slave" does not exist
127.0.98.20:45027 - accepting connections
I20260927 00:43:10.371441 392 mini_ranger.cc:161] Starting Ranger...
I20260927 00:43:10.386108 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 00:43:10.832396 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 00:43:10.832492 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin
2026-09-27 00:43:11,097 [I] DB FLAVOR :POSTGRES
2026-09-27 00:43:11,097 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:43:11,098 [I] Checking connection..
2026-09-27 00:43:11,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 00:43:11,356 [I] Checking connection passed.
2026-09-27 00:43:11,357 [I] --------- Verifying version history table ---------
2026-09-27 00:43:11,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:43:11,590 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:43:11,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:43:11,839 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 00:43:11,839 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 00:43:11,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 00:43:12.091 UTC [12281] WARNING: there is no transaction in progress
2026-09-27 00:43:12,097 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 00:43:12,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 00:43:12,335 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 00:43:12,335 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:43:12,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 00:43:12,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 00:43:12,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pkws.c.gcp-upstream.internal','N') ;"
2026-09-27 00:43:13,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:43:13,309 [I] Table x_portal_user does not exist in database ranger
2026-09-27 00:43:13,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:43:13,561 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 00:43:13,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:43:13,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:43:14,044 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 00:43:14,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 00:43:14.978 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:14.993 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.009 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.023 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.039 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.056 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.108 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.115 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.128 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.138 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.148 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.161 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.170 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.176 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.182 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15.188 UTC [12389] WARNING: there is no transaction in progress
2026-09-27 00:43:15,783 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 00:43:15,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 00:43:16,031 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 00:43:16,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 00:43:16,266 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 00:43:16,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 00:43:16,504 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 00:43:16,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 00:43:16,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 00:43:16,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pkws.c.gcp-upstream.internal';"
2026-09-27 00:43:17,217 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 00:43:17,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 00:43:17,468 [I] DB_PATCHES have already been applied
I20260927 00:43:17.474486 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskql8uh0/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 00:43:17.474553 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 00:43:17.477805 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:35539
Sep 27, 2026 12:43:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:43:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:43:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:43:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:43:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:43:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 00:43:18.576349 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.205s user 0.006s sys 0.003s
Sep 27, 2026 12:43:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:43:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 12:43:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:43:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:43:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 00:43:38.857602 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 00:43:38 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469818, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 00:43:38.938448 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 00:43:38.947705 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 00:43:39.772439 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 00:43:39.772527 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms
I20260927 00:43:39.772552 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45027
2026-09-27 00:43:39,955 [I] DB FLAVOR :POSTGRES
2026-09-27 00:43:39,955 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 00:43:39,955 [I] Checking connection
2026-09-27 00:43:39,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 00:43:40,217 [I] connection success
2026-09-27 00:43:40,217 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 00:43:40,217 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 00:43:40,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 00:43:40,438 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 00:43:40,438 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 00:43:40,438 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 00:43:40,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45027/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 00:43:40,689 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 00:43:41.125898 392 mini_ranger_kms.cc:310] Created kms service
I20260927 00:43:41.337993 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 00:43:41.379844 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 00:43:41 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469818, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:41.757200 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 00:43:41.757457 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskql8uh0/build/debug/bin/postgresql.jar:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskql8uh0/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 00:43:41.757501 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 00:43:41.761535 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12569
I20260927 00:43:41.761617 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59827
Sep 27, 2026 12:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:43:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:43:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:43:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 00:43:43 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:43.995245 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.057s user 0.017s sys 0.149s
I20260927 00:43:43.995383 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 00:43:43.995483 392 mini_ranger_kms.cc:401] 127.0.98.20:59827/kms/v1/keys
Sep 27 00:43:44 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:43:44 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469823, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 00:43:46 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:46.975899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33659
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:47.092610 12620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:47.092849 12620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:47.092942 12620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:47.097285 12620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:47.097388 12620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:47.097442 12620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:47.097476 12620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:47.097498 12620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:47.106146 12620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33659
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:47.107189 12620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:47.108187 12620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:47.116823 12625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:47.116870 12626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:47.117141 12628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:47.117123 12620 server_base.cc:1034] running on GCE node
I20260927 00:43:47.117715 12620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:47.118741 12620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:47.119923 12620 hybrid_clock.cc:648] HybridClock initialized: now 1790469827119902 us; error 40 us; skew 500 ppm
Sep 27 00:43:47 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:47.126448 12620 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:47.127427 12620 webserver.cc:460] Webserver started at http://127.0.98.62:44063/ using document root <none> and password file <none>
I20260927 00:43:47.127960 12620 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:47.128041 12620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:47.128262 12620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:47 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:47.174998 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "3a44c68cfcad40889611e8ec6a515706"
format_stamp: "Formatted at 2026-09-27 00:43:47 on dist-test-slave-pkws"
server_key: "4277e8092e9d08b13d6231c78fcd4f81"
server_key_iv: "2872448ffce961f07a7375d6b36efd8a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:47.175693 12620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a44c68cfcad40889611e8ec6a515706"
format_stamp: "Formatted at 2026-09-27 00:43:47 on dist-test-slave-pkws"
server_key: "4277e8092e9d08b13d6231c78fcd4f81"
server_key_iv: "2872448ffce961f07a7375d6b36efd8a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:47.180423 12620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 00:43:47.193603 12635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:47.195007 12620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:47.195168 12620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "3a44c68cfcad40889611e8ec6a515706"
format_stamp: "Formatted at 2026-09-27 00:43:47 on dist-test-slave-pkws"
server_key: "4277e8092e9d08b13d6231c78fcd4f81"
server_key_iv: "2872448ffce961f07a7375d6b36efd8a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:47.195294 12620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:47.209746 12620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:47.212707 12620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:47.212953 12620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:47.220824 12620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33659
I20260927 00:43:47.220827 12687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33659 every 8 connection(s)
I20260927 00:43:47.221873 12620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:47.223372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 12620
I20260927 00:43:47.223505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:47.223842 392 external_mini_cluster.cc:1428] Setting key 685dc22304b7229b17481beda5e765ab
I20260927 00:43:47.225312 12688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:47.232136 12688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Bootstrap starting.
Sep 27 00:43:47 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:47.234647 12688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:47.235962 12688 log.cc:824] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:47.238195 12688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: No bootstrap required, opened a new log
I20260927 00:43:47.238231 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:47.225300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.0.98.62:33659)
0927 00:43:47.225730 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:47.225748 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:47.225793 (+ 45us) server_negotiation.cc:407] Connection header received
0927 00:43:47.226409 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:47.226424 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:47.226736 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:47.227061 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:47.227861 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:47.229435 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:47.230312 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:47.230773 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:47.230990 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:47.233653 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:47.233658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:47.233671 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:47.233696 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:47.235337 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:47.235771 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:47.235777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:47.235783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:47.235891 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:47.236224 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:47.236228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:47.236229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:47.236728 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:47.236921 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:47.237176 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:43:47.237552 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":143,"threads_started":1}
I20260927 00:43:47.241791 12688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:47.242076 12688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:47.242131 12688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a44c68cfcad40889611e8ec6a515706, State: Initialized, Role: FOLLOWER
I20260927 00:43:47.242576 12688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:47.242731 12688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:47.242790 12688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:47.242883 12688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:47.243897 12688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:47.244329 12688 leader_election.cc:302] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a44c68cfcad40889611e8ec6a515706; no voters:
I20260927 00:43:47.244681 12688 leader_election.cc:288] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:47.244846 12693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:47.245147 12693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 LEADER]: Becoming Leader. State: Replica: 3a44c68cfcad40889611e8ec6a515706, State: Running, Role: LEADER
I20260927 00:43:47.245465 12693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:47.246039 12688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:47.246482 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a44c68cfcad40889611e8ec6a515706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } } }
I20260927 00:43:47.246636 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:47.246829 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a44c68cfcad40889611e8ec6a515706. Latest consensus state: current_term: 1 leader_uuid: "3a44c68cfcad40889611e8ec6a515706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } } }
I20260927 00:43:47.246909 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:47.247159 12698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:47.250176 12698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:47.250675 12688 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:43:47.262599 12698 catalog_manager.cc:1347] Generated new cluster ID: 9e761c285cf84069a2f551109a2b0d07
I20260927 00:43:47.262679 12698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:47.278959 12698 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:47.280197 12698 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:47.294378 12698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Generated new TSK 0
I20260927 00:43:47.295172 12698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:43:48.008348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:48.150353 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:48.150669 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:48.150734 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:48.155995 12733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:48.156076 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:48.156173 12733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:48.161499 12733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:48.162474 12733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:48.163451 12733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:48.173949 12739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.173998 12741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.174352 12738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:48.174381 12733 server_base.cc:1034] running on GCE node
I20260927 00:43:48.175158 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:48.175894 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:48.179203 12733 hybrid_clock.cc:648] HybridClock initialized: now 1790469828179166 us; error 49 us; skew 500 ppm
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:48.182796 12733 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:48.183782 12733 webserver.cc:460] Webserver started at http://127.0.98.1:39587/ using document root <none> and password file <none>
I20260927 00:43:48.184468 12733 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:48.184557 12733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:48.184798 12733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:48.202100 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "84a3a738003b4cd0982bfd4244a56435"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "378d019c1521e5b4443ed7ac4e4ae784"
server_key_iv: "6c31e1d165d7244c7b1269a54900b4a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.202782 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84a3a738003b4cd0982bfd4244a56435"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "378d019c1521e5b4443ed7ac4e4ae784"
server_key_iv: "6c31e1d165d7244c7b1269a54900b4a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.207856 12733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 00:43:48.222966 12750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.226246 12733 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 00:43:48.226438 12733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "84a3a738003b4cd0982bfd4244a56435"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "378d019c1521e5b4443ed7ac4e4ae784"
server_key_iv: "6c31e1d165d7244c7b1269a54900b4a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.226644 12733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:48.245920 12733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:48.249337 12733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:48.249559 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:48.250249 12733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:48.251277 12733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:48.251380 12733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.251503 12733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:48.251601 12733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.261852 12733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37377
I20260927 00:43:48.262974 12733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:48.268078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 12733
I20260927 00:43:48.268190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:48.268481 392 external_mini_cluster.cc:1428] Setting key 1da72bb63f0bcf9e6e14fd866460cdae
I20260927 00:43:48.272204 12863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37377 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:43:48.294871 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.274372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41725 (local address 127.0.98.62:33659)
0927 00:43:48.274681 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:48.274691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:48.275580 (+ 889us) server_negotiation.cc:407] Connection header received
0927 00:43:48.279832 (+ 4252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:48.279837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:48.279903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:48.279973 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:48.281777 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.282628 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.284331 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.284681 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.284763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.290350 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:48.290354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:48.290357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:48.290385 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:48.292332 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.293016 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.293019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.293021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.293109 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.293524 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.293527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.293529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.293741 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:48.293819 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:48.294547 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 00:43:48.294686 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":111,"threads_started":1}
I20260927 00:43:48.296793 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.274842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.1:41725)
0927 00:43:48.275418 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:48.275479 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:48.276624 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:48.280505 (+ 3881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:48.280517 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:48.280869 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:48.281592 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.281617 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.282785 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.282789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.284177 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.284188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.284996 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.285000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.285196 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.285894 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:48.285915 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:48.290129 (+ 4214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:48.292475 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.292481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.292498 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.292875 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.293225 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.293228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.293230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.293416 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.293886 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:48.293893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:48.294244 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 00:43:48.295369 (+ 1125us) client_negotiation.cc:240] Negotiation successful
0927 00:43:48.295711 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":152,"threads_started":1}
I20260927 00:43:48.298558 12864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:48.298791 12864 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:48.299358 12864 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:48.301168 12652 ts_manager.cc:194] Registered new tserver with Master: 84a3a738003b4cd0982bfd4244a56435 (127.0.98.1:37377)
I20260927 00:43:48.302740 12652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:43:48.320261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 00:43:48.475111 12872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:48.475461 12872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:48.475534 12872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:48.481603 12872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:48.481673 12872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:48.481761 12872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:48.489548 12872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:48.490854 12872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:48.492180 12872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:48.504282 12878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.504580 12879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.504344 12881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:48.505342 12872 server_base.cc:1034] running on GCE node
I20260927 00:43:48.505910 12872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:48.506652 12872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:48.507855 12872 hybrid_clock.cc:648] HybridClock initialized: now 1790469828507787 us; error 83 us; skew 500 ppm
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:48.511212 12872 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:48.512014 12872 webserver.cc:460] Webserver started at http://127.0.98.2:40737/ using document root <none> and password file <none>
I20260927 00:43:48.512563 12872 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:48.512621 12872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:48.512847 12872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:48.527441 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "5eaaf79121564cd0af101242de96add0"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "853a78135a5d29b1a7aef009c218325d"
server_key_iv: "1f678cd52eba975970fa2d5c54015ed4"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.528138 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5eaaf79121564cd0af101242de96add0"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "853a78135a5d29b1a7aef009c218325d"
server_key_iv: "1f678cd52eba975970fa2d5c54015ed4"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.538580 12872 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
I20260927 00:43:48.557574 12888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.559134 12872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 00:43:48.559274 12872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "5eaaf79121564cd0af101242de96add0"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "853a78135a5d29b1a7aef009c218325d"
server_key_iv: "1f678cd52eba975970fa2d5c54015ed4"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.559422 12872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:48.597692 12872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:48.601613 12872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:48.601816 12872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:48.602531 12872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:48.603581 12872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:48.603659 12872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.603730 12872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:48.603785 12872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.615631 12872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44037
I20260927 00:43:48.615793 13001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44037 every 8 connection(s)
I20260927 00:43:48.616846 12872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:48.619444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 12872
I20260927 00:43:48.619570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:48.619886 392 external_mini_cluster.cc:1428] Setting key af1052397077039b8d84da23e8321877
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:43:48.632669 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.619534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36751 (local address 127.0.98.62:33659)
0927 00:43:48.619708 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:48.619713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:48.621761 (+ 2048us) server_negotiation.cc:407] Connection header received
0927 00:43:48.623003 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:48.623014 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:48.623081 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:48.623174 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:48.624407 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.625027 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.626251 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.626510 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.626626 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.629028 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:48.629031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:48.629033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:48.629063 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:48.630493 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.630992 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.630997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.630999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.631134 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.631536 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.631540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.631542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.631716 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:48.631823 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:48.632360 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 00:43:48.632496 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 00:43:48.633496 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.620090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.2:36751)
0927 00:43:48.621555 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:48.621638 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:48.622737 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:48.623480 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:48.623488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:48.623771 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:48.624256 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.624269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.625149 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.625153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.626112 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.626129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.626659 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.626663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.626873 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.627488 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:48.627503 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:48.628837 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:48.630628 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.630633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.630646 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.630890 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.631249 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.631252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.631253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.631374 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.631820 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:48.631826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:48.632082 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 00:43:48.632279 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 00:43:48.632570 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":142,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:43:48.635035 13002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:48.635315 13002 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:48.635936 13002 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:48.637104 12652 ts_manager.cc:194] Registered new tserver with Master: 5eaaf79121564cd0af101242de96add0 (127.0.98.2:44037)
I20260927 00:43:48.637667 12652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:43:48.654295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 00:43:48.819483 13007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:48.819808 13007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:48.819864 13007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:48.825804 13007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:48.825877 13007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:48.825958 13007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:48.833424 13007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:48.834683 13007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:48.835906 13007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:48.848217 13013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.848456 13015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:48.848990 13012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:48.849411 13007 server_base.cc:1034] running on GCE node
I20260927 00:43:48.849848 13007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:48.850581 13007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:48.851758 13007 hybrid_clock.cc:648] HybridClock initialized: now 1790469828851691 us; error 80 us; skew 500 ppm
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:48.855340 13007 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:48.856460 13007 webserver.cc:460] Webserver started at http://127.0.98.3:33929/ using document root <none> and password file <none>
I20260927 00:43:48.857211 13007 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:48.857299 13007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:48.857563 13007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:48.872996 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "14b27bc5c09f440e86d345796aa0968c"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "811dc1e6db81059ba2a189cf9b86d173"
server_key_iv: "679bd023a640e945f9b1b9c11354f9a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.873713 13007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14b27bc5c09f440e86d345796aa0968c"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "811dc1e6db81059ba2a189cf9b86d173"
server_key_iv: "679bd023a640e945f9b1b9c11354f9a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.882325 13007 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.002s
I20260927 00:43:48.901294 13022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.903229 13007 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.003s
I20260927 00:43:48.903357 13007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "14b27bc5c09f440e86d345796aa0968c"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "811dc1e6db81059ba2a189cf9b86d173"
server_key_iv: "679bd023a640e945f9b1b9c11354f9a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:48.903477 13007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:48.932535 13007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:48.936276 13007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:48.936447 13007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:48.937080 13007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:48.938056 13007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:48.938113 13007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.938161 13007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:48.938189 13007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:48.948581 13007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36053
I20260927 00:43:48.949664 13007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:48.950987 13135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36053 every 8 connection(s)
I20260927 00:43:48.957224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13007
I20260927 00:43:48.957366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:48.957701 392 external_mini_cluster.cc:1428] Setting key ab37ebccf1ab2fb1888ba3e5b1acfb59
Sep 27 00:43:48 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:48.968014 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.952687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40527 (local address 127.0.98.62:33659)
0927 00:43:48.952813 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:48.952817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:48.953906 (+ 1089us) server_negotiation.cc:407] Connection header received
0927 00:43:48.955215 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:48.955219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:48.955271 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:48.955342 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:48.956799 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.957685 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.959570 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.959937 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.960039 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.963432 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:48.963436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:48.963439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:48.963467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:48.965662 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.966295 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.966298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.966300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.966387 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:48.966778 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:48.966781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:48.966783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:48.966985 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:48.967096 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:48.967738 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 00:43:48.967857 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:48.969213 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:48.953211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.3:40527)
0927 00:43:48.953750 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:48.953806 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:48.954840 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:48.955640 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:48.955649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:48.955983 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:48.956636 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.956653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.957843 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.957847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.959397 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:48.959407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:48.960566 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:48.960569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:48.960767 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:48.961474 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:48.961494 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:48.963260 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:48.965788 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.965795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.965812 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.966186 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.966488 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:48.966491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:48.966493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:48.966675 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:48.967110 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:48.967117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:48.967455 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 00:43:48.967682 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 00:43:48.968029 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260927 00:43:48.970374 13136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:48.970641 13136 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:48.971226 13136 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:48.972335 12651 ts_manager.cc:194] Registered new tserver with Master: 14b27bc5c09f440e86d345796aa0968c (127.0.98.3:36053)
I20260927 00:43:48.973138 12651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40527
I20260927 00:43:48.974030 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:48.974975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 12733
I20260927 00:43:48.982936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 12872
I20260927 00:43:48.989275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13007
I20260927 00:43:48.996495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 12620
I20260927 00:43:49.005486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33659
--webserver_interface=127.0.98.62
--webserver_port=44063
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33659
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:49.122575 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:49.122835 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:49.122875 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:49.126823 13139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:49.126876 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:49.126895 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:49.126911 13139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:49.126926 13139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:49.132318 13139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33659
--ranger_config_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:49.133409 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:49.134443 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:49.142978 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.142977 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:49.143117 13139 server_base.cc:1034] running on GCE node
W20260927 00:43:49.142977 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:49.143869 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:49.144965 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:49.146129 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790469829146117 us; error 28 us; skew 500 ppm
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:49.149082 13139 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:49.149909 13139 webserver.cc:460] Webserver started at http://127.0.98.62:44063/ using document root <none> and password file <none>
I20260927 00:43:49.150403 13139 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:49.150525 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:49.212481 13139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:49.214999 13155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.216329 13139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:49.216475 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "3a44c68cfcad40889611e8ec6a515706"
format_stamp: "Formatted at 2026-09-27 00:43:47 on dist-test-slave-pkws"
server_key: "4277e8092e9d08b13d6231c78fcd4f81"
server_key_iv: "2872448ffce961f07a7375d6b36efd8a"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:49.217118 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:49.241194 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:49.244344 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:49.244591 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:49.251993 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33659 every 8 connection(s)
I20260927 00:43:49.251997 13139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33659
I20260927 00:43:49.253109 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:49.257200 13208 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:43:49.258720 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Bootstrap starting.
I20260927 00:43:49.262842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13139
I20260927 00:43:49.263315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33659
--webserver_port=39587
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:49.275852 13208 log.cc:824] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:49.282483 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:43:49.282871 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Bootstrap complete.
I20260927 00:43:49.286942 13208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:49.287357 13208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a44c68cfcad40889611e8ec6a515706, State: Initialized, Role: FOLLOWER
I20260927 00:43:49.287871 13208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:49.287982 13208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:49.288065 13208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:49.288187 13208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:43:49.290050 13208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:49.290373 13208 leader_election.cc:302] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a44c68cfcad40889611e8ec6a515706; no voters:
I20260927 00:43:49.290701 13208 leader_election.cc:288] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:43:49.290781 13212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:43:49.291030 13212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [term 2 LEADER]: Becoming Leader. State: Replica: 3a44c68cfcad40889611e8ec6a515706, State: Running, Role: LEADER
I20260927 00:43:49.291378 13212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } }
I20260927 00:43:49.291903 13208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:49.292651 13213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a44c68cfcad40889611e8ec6a515706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } } }
I20260927 00:43:49.292769 13213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:49.293004 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a44c68cfcad40889611e8ec6a515706. Latest consensus state: current_term: 2 leader_uuid: "3a44c68cfcad40889611e8ec6a515706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a44c68cfcad40889611e8ec6a515706" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33659 } } }
I20260927 00:43:49.293079 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:49.293515 13222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:49.296784 13222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:49.297439 13222 catalog_manager.cc:1259] Loaded cluster ID: 9e761c285cf84069a2f551109a2b0d07
I20260927 00:43:49.297569 13222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:49.299909 13222 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:49.300572 13222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Loaded TSK: 0
I20260927 00:43:49.297361 13208 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 00:43:49.301795 13222 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:43:49.385254 13210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:49.385624 13210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:49.385710 13210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:49.390214 13210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:49.390282 13210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:49.390378 13210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:49.396412 13210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39587
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:49.397717 13210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:49.398983 13210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:49.408988 13248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.409327 13246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.411942 13245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:49.412320 13210 server_base.cc:1034] running on GCE node
I20260927 00:43:49.412858 13210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:49.413683 13210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:49.414881 13210 hybrid_clock.cc:648] HybridClock initialized: now 1790469829414836 us; error 54 us; skew 500 ppm
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:49.418118 13210 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:49.419281 13210 webserver.cc:460] Webserver started at http://127.0.98.1:39587/ using document root <none> and password file <none>
I20260927 00:43:49.420037 13210 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:49.420151 13210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:49.447124 13210 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260927 00:43:49.450484 13256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.452076 13210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:43:49.452214 13210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "84a3a738003b4cd0982bfd4244a56435"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "378d019c1521e5b4443ed7ac4e4ae784"
server_key_iv: "6c31e1d165d7244c7b1269a54900b4a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:49.453054 13210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:49.487254 13210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:49.490898 13210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:49.491096 13210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:49.491798 13210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:49.493067 13210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:49.493134 13210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.493244 13210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:49.493280 13210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.503968 13210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37377
I20260927 00:43:49.505231 13210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:49.506816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13210
I20260927 00:43:49.507234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44037
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33659
--webserver_port=40737
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:49.511380 13369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37377 every 8 connection(s)
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:49.538316 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:49.514720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.1:51201)
0927 00:43:49.521237 (+ 6517us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:49.521362 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:49.522709 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:49.524529 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:49.524537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:49.524874 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:49.525482 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:49.525498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.527338 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.527342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:49.528674 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:49.528684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.529879 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.529883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:49.530112 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:49.530786 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:49.530805 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:49.532525 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:49.534936 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:49.534942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:49.534955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:49.535268 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:49.535645 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:49.535648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:49.535650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:49.535815 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:49.536550 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:49.536556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:49.536848 (+ 292us) client_negotiation.cc:769] Sending connection context
0927 00:43:49.537068 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 00:43:49.537376 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6252,"thread_start_us":150,"threads_started":1}
I20260927 00:43:49.539760 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:49.516221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51201 (local address 127.0.98.62:33659)
0927 00:43:49.517034 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:49.517060 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:49.523036 (+ 5976us) server_negotiation.cc:407] Connection header received
0927 00:43:49.523330 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:49.523343 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:49.523746 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:49.524062 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:49.525675 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.527192 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.528807 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.529515 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.529670 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:49.532742 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:49.532749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:49.532764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:49.532793 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:49.534797 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:49.535395 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:49.535400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:49.535406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:49.535535 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:49.535922 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:49.535927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:49.535930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:49.536374 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:49.536497 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:49.538665 (+ 2168us) server_negotiation.cc:299] Negotiation successful
0927 00:43:49.538931 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":113,"threads_started":1}
I20260927 00:43:49.542800 13370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:49.543053 13370 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.543606 13370 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:49.545841 13370 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33659 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:43:49.545951 13370 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.546098 13370 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:49.546939 13370 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.547096 13370 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:49.547827 13370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:43:49.651410 13372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:49.651739 13372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:49.651823 13372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:49.657922 13372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:49.658010 13372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:49.658140 13372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:49.664004 13372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40737
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:49.665269 13372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:49.666348 13372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:49.675994 13380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.676026 13379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.676299 13382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:49.676853 13372 server_base.cc:1034] running on GCE node
I20260927 00:43:49.677270 13372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:49.677980 13372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:49.679157 13372 hybrid_clock.cc:648] HybridClock initialized: now 1790469829679124 us; error 45 us; skew 500 ppm
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:49.682160 13372 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:49.683094 13372 webserver.cc:460] Webserver started at http://127.0.98.2:40737/ using document root <none> and password file <none>
I20260927 00:43:49.683770 13372 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:49.683874 13372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:49.700248 13372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:43:49.704142 13390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.705878 13372 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 00:43:49.706020 13372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "5eaaf79121564cd0af101242de96add0"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "853a78135a5d29b1a7aef009c218325d"
server_key_iv: "1f678cd52eba975970fa2d5c54015ed4"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:49.706760 13372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:49.742414 13372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:49.746001 13372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:49.746201 13372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:49.746903 13372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:49.748225 13372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:49.748289 13372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.748349 13372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:49.748384 13372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.759341 13372 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44037
I20260927 00:43:49.760628 13372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:49.761010 13503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44037 every 8 connection(s)
I20260927 00:43:49.770634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13372
I20260927 00:43:49.771142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36053
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33659
--webserver_port=33929
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:49.778589 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:49.762926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59589 (local address 127.0.98.62:33659)
0927 00:43:49.763060 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:49.763065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:49.764196 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 00:43:49.765851 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:49.765855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:49.765917 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:49.766000 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:49.767076 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.767654 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.769239 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.769495 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.769611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:49.773845 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:49.773850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:49.773856 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:49.773892 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:49.775761 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:49.776413 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:49.776419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:49.776424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:49.776517 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:49.776994 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:49.776999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:49.777003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:49.777284 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:49.777382 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:49.778174 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 00:43:49.778342 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:49.781004 13506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:49.763460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.2:59589)
0927 00:43:49.764028 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:49.764084 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:49.765632 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:49.766178 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:49.766184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:49.766453 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:49.766921 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:49.766935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.767795 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.767801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:49.769095 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:49.769104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:49.769652 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:49.769656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:49.771060 (+ 1404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:49.771776 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:49.771804 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:49.773631 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:49.775895 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:49.775903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:49.775923 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:49.776294 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:49.776632 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:49.776637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:49.776643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:49.776837 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:49.777477 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:49.777486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:49.777854 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 00:43:49.779109 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0927 00:43:49.779517 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":160,"threads_started":1}
I20260927 00:43:49.782940 13504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:49.783236 13504 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.783913 13504 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:49.789014 13504 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33659 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:43:49.789151 13504 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.789346 13504 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:49.790098 13504 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:49.790277 13504 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:49.790946 13504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:43:49.915994 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:49.916296 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:49.916378 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:49.921275 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:49.921355 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:49.921442 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:49.929191 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33929
--tserver_master_addrs=127.0.98.62:33659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:49.930454 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:49.931720 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:49.944016 13513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.943982 13514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:49.944154 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:49.944453 13507 server_base.cc:1034] running on GCE node
I20260927 00:43:49.944952 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:49.945649 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:49.946841 13507 hybrid_clock.cc:648] HybridClock initialized: now 1790469829946817 us; error 37 us; skew 500 ppm
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:49.950073 13507 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:49.951162 13507 webserver.cc:460] Webserver started at http://127.0.98.3:33929/ using document root <none> and password file <none>
I20260927 00:43:49.951857 13507 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:49.952035 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 00:43:49 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:49.970271 13507 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260927 00:43:49.978032 13524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:49.979619 13507 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260927 00:43:49.979712 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "14b27bc5c09f440e86d345796aa0968c"
format_stamp: "Formatted at 2026-09-27 00:43:48 on dist-test-slave-pkws"
server_key: "811dc1e6db81059ba2a189cf9b86d173"
server_key_iv: "679bd023a640e945f9b1b9c11354f9a5"
server_key_version: "kuduclusterkey@0"
I20260927 00:43:49.980221 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:49.994724 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:49.998172 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:49.998355 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:49.999006 13507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:50.000005 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:50.000064 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:50.000114 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:50.000144 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:50.020372 13507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36053
I20260927 00:43:50.027300 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36053 every 8 connection(s)
I20260927 00:43:50.028232 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:50.032871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13507
Sep 27 00:43:50 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:50.045596 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:50.035957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.0.98.62:33659)
0927 00:43:50.036299 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:50.036309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:50.036324 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:43:50.036372 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:50.036376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:50.036424 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:50.037365 (+ 941us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:50.037530 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.038094 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.039012 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.039358 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.039460 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:50.041401 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:50.041405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:50.041407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:50.041434 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:50.043349 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:50.043780 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:50.043785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:50.043787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:50.043869 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:50.044291 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:50.044294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:50.044296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:50.044531 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:50.044765 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:50.045157 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 00:43:50.045376 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
I20260927 00:43:50.048681 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:50.031816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50327 (local address 127.0.98.62:33659)
0927 00:43:50.031953 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:50.031958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:50.032265 (+ 307us) server_negotiation.cc:407] Connection header received
0927 00:43:50.037212 (+ 4947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:50.037215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:50.037265 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:50.037322 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:50.039104 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.039744 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.041166 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.041429 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.041562 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:50.044507 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:50.044510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:50.044513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:50.044552 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:50.046318 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:50.046955 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:50.046958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:50.046960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:50.047053 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:50.047458 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:50.047461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:50.047462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:50.047626 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:50.047699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:50.048377 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 00:43:50.048549 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":60}
I20260927 00:43:50.049780 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:50.031040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33659 (local address 127.0.98.3:50327)
0927 00:43:50.032062 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:50.032122 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:50.036942 (+ 4820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:50.037844 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:50.037855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:50.038203 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:50.038905 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:50.038927 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.039864 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.039867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:50.041041 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:50.041051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:50.041748 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:50.041755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:50.041922 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:50.042547 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:50.042571 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:50.044353 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:50.046437 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:50.046443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:50.046459 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:50.046827 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:50.047155 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:50.047158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:50.047160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:50.047357 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:50.047738 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:50.047745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:50.048080 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 00:43:50.048320 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 00:43:50.048670 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":156,"threads_started":1}
I20260927 00:43:50.051090 13638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33659
I20260927 00:43:50.051347 13638 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:50.051957 13638 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:50.053647 13638 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33659 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 00:43:50.053772 13638 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:50.053949 13638 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:50.054759 13638 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:50.054924 13638 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
W20260927 00:43:50.055673 13638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 00:43:50 dist-test-slave-pkws krb5kdc[12176](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790469830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:50.548059 13370 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:50.548383 13370 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:50.549578 13172 ts_manager.cc:194] Registered new tserver with Master: 84a3a738003b4cd0982bfd4244a56435 (127.0.98.1:37377)
I20260927 00:43:50.551297 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51201
Sep 27 00:43:50 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:50.791168 13504 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:50.791482 13504 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:50.792269 13172 ts_manager.cc:194] Registered new tserver with Master: 5eaaf79121564cd0af101242de96add0 (127.0.98.2:44037)
I20260927 00:43:50.793136 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59589
Sep 27 00:43:50 dist-test-slave-pkws krb5kdc[12176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 00:43:51.055909 13638 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:51.056222 13638 heartbeater.cc:511] Master 127.0.98.62:33659 requested a full tablet report, sending...
I20260927 00:43:51.057118 13172 ts_manager.cc:194] Registered new tserver with Master: 14b27bc5c09f440e86d345796aa0968c (127.0.98.3:36053)
I20260927 00:43:51.057878 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50327
I20260927 00:43:51.062705 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:51.086947 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.069380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.0.98.62:33659)
0927 00:43:51.077081 (+ 7701us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.077086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.077571 (+ 485us) server_negotiation.cc:407] Connection header received
0927 00:43:51.077626 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.077629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.077682 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.077767 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:51.078606 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.079443 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.080674 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.081069 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.081178 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.083023 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:51.083027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:51.083030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:51.083058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:51.084943 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:51.085434 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:51.085438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:51.085440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:51.085529 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:51.085849 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:51.085852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:51.085854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:51.086091 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:51.086223 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.086591 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.086735 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7602,"thread_start_us":88,"threads_started":1}
I20260927 00:43:51.091351 13172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:51.096539 13172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:51.128075 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.119746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:42414)
0927 00:43:51.122509 (+ 2763us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.122523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:51.122619 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:51.123309 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:51.123312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:51.123331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:51.123422 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.123427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.124916 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.124920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.126306 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.126317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.127544 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.127547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.127654 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.127689 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:43:51.127791 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:43:51.127901 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2697,"thread_start_us":89,"threads_started":1}
I20260927 00:43:51.128925 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.120123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.0.98.1:37377)
0927 00:43:51.121611 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.121619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.122715 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 00:43:51.122799 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.122805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.123018 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.123160 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:51.123557 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.124763 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.126444 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.127234 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.127331 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.127484 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.128664 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.128773 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1377,"thread_start_us":80,"threads_started":1}
I20260927 00:43:51.131546 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.118937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36053 (local address 127.0.0.1:32836)
0927 00:43:51.122186 (+ 3249us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.122199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:51.122294 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:51.123236 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:51.123243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:51.123267 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:51.123404 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.123412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.125550 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.125553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.126700 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.126711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.131129 (+ 4418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.131133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.131238 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.131261 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:43:51.131336 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:43:51.131405 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3177,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260927 00:43:51.132228 13304 tablet_service.cc:1467] Processing CreateTablet for tablet 4e87abce285841da93d9b1f9114bbe36 (DEFAULT_TABLE table=test-table [id=5fe08a362c9e4ae88da3a5dd4375d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:51.133105 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.119353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.0.98.3:36053)
0927 00:43:51.122536 (+ 3183us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.122543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.122567 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:43:51.122646 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.122652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.122868 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.123023 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:51.123544 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.125412 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.129966 (+ 4554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.130821 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.130926 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.131064 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.132835 (+ 1771us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.132964 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3075,"thread_start_us":63,"threads_started":1}
I20260927 00:43:51.133410 13304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e87abce285841da93d9b1f9114bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:51.136123 13569 tablet_service.cc:1467] Processing CreateTablet for tablet 4e87abce285841da93d9b1f9114bbe36 (DEFAULT_TABLE table=test-table [id=5fe08a362c9e4ae88da3a5dd4375d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:51.137313 13569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e87abce285841da93d9b1f9114bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:51.138913 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.116479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44037 (local address 127.0.0.1:59640)
0927 00:43:51.121781 (+ 5302us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.121835 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:51.121961 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:51.128379 (+ 6418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:51.128382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:51.128400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:51.128480 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.128486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.130814 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.130817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.131876 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.131888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.138498 (+ 6610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.138502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.138591 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.138613 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:43:51.138690 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:43:51.138773 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4897,"thread_start_us":108,"threads_started":1}
I20260927 00:43:51.139370 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.117137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.98.2:44037)
0927 00:43:51.126976 (+ 9839us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.126985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.127008 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:43:51.127080 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.127085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.127288 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.128208 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:51.128616 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.130668 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.137347 (+ 6679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.138186 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.138297 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.138431 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.139124 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.139234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9721,"thread_start_us":91,"threads_started":1}
I20260927 00:43:51.142189 13438 tablet_service.cc:1467] Processing CreateTablet for tablet 4e87abce285841da93d9b1f9114bbe36 (DEFAULT_TABLE table=test-table [id=5fe08a362c9e4ae88da3a5dd4375d7a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:51.143241 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e87abce285841da93d9b1f9114bbe36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:51.158797 13674 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Bootstrap starting.
I20260927 00:43:51.160037 13675 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Bootstrap starting.
I20260927 00:43:51.162344 13675 tablet_bootstrap.cc:654] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:51.163910 13675 log.cc:824] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:51.165723 13674 tablet_bootstrap.cc:654] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:51.166167 13675 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: No bootstrap required, opened a new log
I20260927 00:43:51.166345 13675 ts_tablet_manager.cc:1397] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:43:51.166492 13674 log.cc:824] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:51.169471 13673 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Bootstrap starting.
I20260927 00:43:51.170936 13675 raft_consensus.cc:348] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.171212 13675 raft_consensus.cc:374] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:51.171274 13675 raft_consensus.cc:729] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eaaf79121564cd0af101242de96add0, State: Initialized, Role: FOLLOWER
I20260927 00:43:51.171535 13673 tablet_bootstrap.cc:654] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:51.171973 13675 consensus_queue.cc:260] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.173161 13675 ts_tablet_manager.cc:1428] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 00:43:51.173560 13504 heartbeater.cc:503] Master 127.0.98.62:33659 was elected leader, sending a full tablet report...
I20260927 00:43:51.173928 13673 log.cc:824] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:51.179883 13674 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: No bootstrap required, opened a new log
I20260927 00:43:51.180079 13674 ts_tablet_manager.cc:1397] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20260927 00:43:51.180431 13673 tablet_bootstrap.cc:492] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: No bootstrap required, opened a new log
I20260927 00:43:51.180694 13673 ts_tablet_manager.cc:1397] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260927 00:43:51.184096 13674 raft_consensus.cc:348] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.184360 13674 raft_consensus.cc:374] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:51.184405 13674 raft_consensus.cc:729] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b27bc5c09f440e86d345796aa0968c, State: Initialized, Role: FOLLOWER
I20260927 00:43:51.184832 13674 consensus_queue.cc:260] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.185883 13674 ts_tablet_manager.cc:1428] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 00:43:51.185886 13673 raft_consensus.cc:348] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.186219 13673 raft_consensus.cc:374] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:51.186318 13673 raft_consensus.cc:729] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a3a738003b4cd0982bfd4244a56435, State: Initialized, Role: FOLLOWER
I20260927 00:43:51.186360 13638 heartbeater.cc:503] Master 127.0.98.62:33659 was elected leader, sending a full tablet report...
I20260927 00:43:51.186797 13673 consensus_queue.cc:260] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.187687 13673 ts_tablet_manager.cc:1428] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:43:51.187994 13370 heartbeater.cc:503] Master 127.0.98.62:33659 was elected leader, sending a full tablet report...
W20260927 00:43:51.263792 13505 tablet.cc:2369] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:51.266211 13371 tablet.cc:2369] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:51.280617 13639 tablet.cc:2369] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:51.338229 13681 raft_consensus.cc:482] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:51.338513 13681 raft_consensus.cc:504] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.339715 13681 leader_election.cc:288] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eaaf79121564cd0af101242de96add0 (127.0.98.2:44037), 14b27bc5c09f440e86d345796aa0968c (127.0.98.3:36053)
I20260927 00:43:51.343721 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.340097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38267 (local address 127.0.98.3:36053)
0927 00:43:51.340230 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.340236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.340455 (+ 219us) server_negotiation.cc:407] Connection header received
0927 00:43:51.340662 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.340665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.340716 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.340817 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:51.341129 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.341626 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.342751 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.343150 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.343276 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.343337 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.343524 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.343603 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 00:43:51.343741 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.339973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36053 (local address 127.0.98.1:38267)
0927 00:43:51.340343 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.340354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:51.340539 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:51.340828 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:51.340831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:51.340846 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:51.341005 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.341010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.341737 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.341743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.342629 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.342638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.343278 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.343281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.343376 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.343406 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:43:51.343509 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:43:51.343575 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1,"spinlock_wait_cycles":20352,"thread_start_us":127,"threads_started":1}
I20260927 00:43:51.344233 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.339962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44037 (local address 127.0.98.1:49719)
0927 00:43:51.340334 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.340346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:51.340479 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:51.340757 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:51.340760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:51.340774 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:51.340992 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.340999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.341837 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.341844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.342743 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:51.342752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.343715 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.343718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:51.343872 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.343907 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:43:51.344012 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 00:43:51.344075 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":57,"thread_start_us":119,"threads_started":1}
I20260927 00:43:51.344254 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.340029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49719 (local address 127.0.98.2:44037)
0927 00:43:51.340182 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.340188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.340436 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:43:51.340599 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.340603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.340653 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.340745 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:51.341101 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.341729 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.342874 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.343446 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.343584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.343657 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.344063 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.344145 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:43:51.344337 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e87abce285841da93d9b1f9114bbe36" candidate_uuid: "84a3a738003b4cd0982bfd4244a56435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b27bc5c09f440e86d345796aa0968c" is_pre_election: true
I20260927 00:43:51.344720 13574 raft_consensus.cc:2454] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a3a738003b4cd0982bfd4244a56435 in term 0.
I20260927 00:43:51.345005 13458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e87abce285841da93d9b1f9114bbe36" candidate_uuid: "84a3a738003b4cd0982bfd4244a56435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaaf79121564cd0af101242de96add0" is_pre_election: true
I20260927 00:43:51.345232 13257 leader_election.cc:302] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b27bc5c09f440e86d345796aa0968c, 84a3a738003b4cd0982bfd4244a56435; no voters:
I20260927 00:43:51.345271 13458 raft_consensus.cc:2454] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a3a738003b4cd0982bfd4244a56435 in term 0.
I20260927 00:43:51.345518 13681 raft_consensus.cc:2781] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:51.345638 13681 raft_consensus.cc:482] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:51.345700 13681 raft_consensus.cc:3037] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:51.346697 13681 raft_consensus.cc:504] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.347039 13681 leader_election.cc:288] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [CANDIDATE]: Term 1 election: Requested vote from peers 5eaaf79121564cd0af101242de96add0 (127.0.98.2:44037), 14b27bc5c09f440e86d345796aa0968c (127.0.98.3:36053)
I20260927 00:43:51.347389 13458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e87abce285841da93d9b1f9114bbe36" candidate_uuid: "84a3a738003b4cd0982bfd4244a56435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eaaf79121564cd0af101242de96add0"
I20260927 00:43:51.347389 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e87abce285841da93d9b1f9114bbe36" candidate_uuid: "84a3a738003b4cd0982bfd4244a56435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b27bc5c09f440e86d345796aa0968c"
I20260927 00:43:51.347510 13458 raft_consensus.cc:3037] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:51.347510 13574 raft_consensus.cc:3037] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:51.348723 13574 raft_consensus.cc:2454] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a3a738003b4cd0982bfd4244a56435 in term 1.
I20260927 00:43:51.348723 13458 raft_consensus.cc:2454] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a3a738003b4cd0982bfd4244a56435 in term 1.
I20260927 00:43:51.349020 13258 leader_election.cc:302] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eaaf79121564cd0af101242de96add0, 84a3a738003b4cd0982bfd4244a56435; no voters:
I20260927 00:43:51.349176 13681 raft_consensus.cc:2781] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:51.349411 13681 raft_consensus.cc:686] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 1 LEADER]: Becoming Leader. State: Replica: 84a3a738003b4cd0982bfd4244a56435, State: Running, Role: LEADER
I20260927 00:43:51.349772 13681 consensus_queue.cc:237] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } }
I20260927 00:43:51.352829 13172 catalog_manager.cc:5495] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84a3a738003b4cd0982bfd4244a56435 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "84a3a738003b4cd0982bfd4244a56435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84a3a738003b4cd0982bfd4244a56435" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 } health_report { overall_health: UNKNOWN } } }
I20260927 00:43:51.414098 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.410244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.0.98.1:37377)
0927 00:43:51.410480 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:51.410486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:51.410544 (+ 58us) server_negotiation.cc:407] Connection header received
0927 00:43:51.410738 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:51.410741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:51.410797 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:51.410912 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:51.411159 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.411668 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.412584 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:51.412847 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:51.412989 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:51.413258 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:51.413708 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:51.413800 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:51.413858 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:43:51.413905 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 00:43:51.419770 13458 raft_consensus.cc:1261] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 1 FOLLOWER]: Refusing update from remote peer 84a3a738003b4cd0982bfd4244a56435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:51.419771 13574 raft_consensus.cc:1261] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 1 FOLLOWER]: Refusing update from remote peer 84a3a738003b4cd0982bfd4244a56435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:51.420663 13691 consensus_queue.cc:1035] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b27bc5c09f440e86d345796aa0968c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:51.420953 13681 consensus_queue.cc:1035] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eaaf79121564cd0af101242de96add0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:51.427445 13694 mvcc.cc:205] Tried to move back new op lower bound from 7333764429488099328 to 7333764429211594752. Current Snapshot: MvccSnapshot[applied={T|T < 7333764429488099328}]
I20260927 00:43:51.429459 13695 mvcc.cc:205] Tried to move back new op lower bound from 7333764429488099328 to 7333764429211594752. Current Snapshot: MvccSnapshot[applied={T|T < 7333764429488099328 or (T in {7333764429488099328})}]
I20260927 00:43:51.436553 13171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:51.436720 13171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:51.438962 13171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a44c68cfcad40889611e8ec6a515706: Sending DeleteTablet for 3 replicas of tablet 4e87abce285841da93d9b1f9114bbe36
I20260927 00:43:51.439666 13304 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e87abce285841da93d9b1f9114bbe36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42414
I20260927 00:43:51.439671 13569 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e87abce285841da93d9b1f9114bbe36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32836
I20260927 00:43:51.440135 13705 tablet_replica.cc:330] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435: stopping tablet replica
I20260927 00:43:51.440147 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e87abce285841da93d9b1f9114bbe36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59640
I20260927 00:43:51.440433 13705 raft_consensus.cc:2229] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:43:51.440712 13706 tablet_replica.cc:330] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: stopping tablet replica
I20260927 00:43:51.440773 13705 raft_consensus.cc:2258] T 4e87abce285841da93d9b1f9114bbe36 P 84a3a738003b4cd0982bfd4244a56435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:51.441202 13706 raft_consensus.cc:2229] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:51.441231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13210
I20260927 00:43:51.441985 13704 tablet_replica.cc:330] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: stopping tablet replica
I20260927 00:43:51.442166 13706 raft_consensus.cc:2258] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:51.442274 13704 raft_consensus.cc:2229] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:51.442559 13704 raft_consensus.cc:2258] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:51.443703 13706 ts_tablet_manager.cc:1905] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:51.444470 13704 ts_tablet_manager.cc:1905] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:51.446578 13704 ts_tablet_manager.cc:1918] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:51.446686 13704 log.cc:1196] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/4e87abce285841da93d9b1f9114bbe36
I20260927 00:43:51.446978 13704 ts_tablet_manager.cc:1939] T 4e87abce285841da93d9b1f9114bbe36 P 14b27bc5c09f440e86d345796aa0968c: Deleting consensus metadata
I20260927 00:43:51.447124 13706 ts_tablet_manager.cc:1918] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:51.447233 13706 log.cc:1196] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/4e87abce285841da93d9b1f9114bbe36
I20260927 00:43:51.447546 13706 ts_tablet_manager.cc:1939] T 4e87abce285841da93d9b1f9114bbe36 P 5eaaf79121564cd0af101242de96add0: Deleting consensus metadata
I20260927 00:43:51.448240 13156 catalog_manager.cc:4841] TS 14b27bc5c09f440e86d345796aa0968c (127.0.98.3:36053): tablet 4e87abce285841da93d9b1f9114bbe36 (table test-table [id=5fe08a362c9e4ae88da3a5dd4375d7a1]) successfully deleted
W20260927 00:43:51.448866 13157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37377 (error 108)
I20260927 00:43:51.449154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13372
I20260927 00:43:51.449358 13157 catalog_manager.cc:4841] TS 5eaaf79121564cd0af101242de96add0 (127.0.98.2:44037): tablet 4e87abce285841da93d9b1f9114bbe36 (table test-table [id=5fe08a362c9e4ae88da3a5dd4375d7a1]) successfully deleted
I20260927 00:43:51.450006 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:51.449639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37377 (local address 127.0.0.1:42442)
0927 00:43:51.449815 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:51.449886 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 00:43:51.450158 13157 catalog_manager.cc:4584] TS 84a3a738003b4cd0982bfd4244a56435 (127.0.98.1:37377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e87abce285841da93d9b1f9114bbe36: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37377: connect: Connection refused (error 111)
I20260927 00:43:51.454645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13507
I20260927 00:43:51.459941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13139
I20260927 00:43:51.466600 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 00:43:51.800752 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 00:43:51.800859 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 00:43:52.145262 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 00:43:52.145 UTC [12206] LOG: received smart shutdown request
2026-09-27 00:43:52.149 UTC [12206] LOG: background worker "logical replication launcher" (PID 12215) exited with exit code 1
2026-09-27 00:43:52.149 UTC [12210] LOG: shutting down
2026-09-27 00:43:52.304 UTC [12206] LOG: database system is shut down
2026-09-27T00:43:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46702 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:52 dist-test-slave-pkws krb5kdc[13716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:52 dist-test-slave-pkws krb5kdc[13716](info): set up 2 sockets
Sep 27 00:43:52 dist-test-slave-pkws krb5kdc[13716](info): commencing operation
krb5kdc: starting...
W20260927 00:43:53.469055 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:53.584378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:53.690904 13729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:53.691174 13729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:53.691252 13729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:53.695152 13729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:53.695247 13729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:53.695288 13729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:53.695341 13729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:53.695398 13729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:53.700397 13729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:53.701398 13729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:53.702339 13729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:53.710664 13735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:53.710662 13734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:53.710665 13737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:53.710974 13729 server_base.cc:1034] running on GCE node
I20260927 00:43:53.711443 13729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:53.712457 13729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:53.713622 13729 hybrid_clock.cc:648] HybridClock initialized: now 1790469833713611 us; error 25 us; skew 500 ppm
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:53.716454 13729 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:53.717643 13729 webserver.cc:460] Webserver started at http://127.0.98.62:41319/ using document root <none> and password file <none>
I20260927 00:43:53.718165 13729 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:53.718225 13729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:53.718459 13729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:53.720466 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "ef229d8c84af407badf53b7f94a834f2"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "a21362017f11ab006ef563b6d02a076e"
server_key_iv: "424a14eaa0c4fc6b27a2d631006f1b7f"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.721092 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef229d8c84af407badf53b7f94a834f2"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "a21362017f11ab006ef563b6d02a076e"
server_key_iv: "424a14eaa0c4fc6b27a2d631006f1b7f"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.724464 13729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:53.726780 13744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:53.727945 13729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:53.728082 13729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "ef229d8c84af407badf53b7f94a834f2"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "a21362017f11ab006ef563b6d02a076e"
server_key_iv: "424a14eaa0c4fc6b27a2d631006f1b7f"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.728186 13729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:53.740983 13729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:53.744004 13729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:53.744205 13729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:53.751327 13729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46181
I20260927 00:43:53.751341 13796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46181 every 8 connection(s)
I20260927 00:43:53.752209 13729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:53.754961 13797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:53.758553 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:53.759728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13729
I20260927 00:43:53.759886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:53.760149 392 external_mini_cluster.cc:1428] Setting key 8839482b553b812a44df499cfa002d44
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:53.777516 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.760920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46181 (local address 127.0.0.1:34612)
0927 00:43:53.761468 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.761523 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:53.762353 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:53.762998 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:53.763002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:53.763429 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:53.763649 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.763662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.765547 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.765550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.766278 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.766288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.767231 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.767234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.767427 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.768027 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:53.768043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:53.771641 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:53.774025 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:53.774033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:53.774040 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:53.774269 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:53.774720 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:53.774724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:53.774726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:53.774884 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:53.775735 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:53.775741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:53.775927 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 00:43:53.776154 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 00:43:53.776481 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":2,"thread_start_us":138,"threads_started":1}
I20260927 00:43:53.777516 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.760920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.0.98.62:46181)
0927 00:43:53.761429 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:53.761454 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:53.761772 (+ 318us) server_negotiation.cc:407] Connection header received
0927 00:43:53.762646 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:53.762650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:53.762813 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:53.763001 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:53.763859 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.765410 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.766450 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.767073 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.767320 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.771793 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:53.771799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:53.771809 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:53.771831 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:53.773898 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:53.774390 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:53.774399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:53.774404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:53.774548 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:53.775000 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:53.775006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:53.775008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:53.775629 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:53.775766 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:53.776224 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 00:43:53.776535 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":28928,"thread_start_us":139,"threads_started":1}
I20260927 00:43:53.780257 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:53.781054 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.780760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43302)
0927 00:43:53.780880 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.780995 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:43:53.781234 13748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
I20260927 00:43:53.782063 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.781825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43306)
0927 00:43:53.781930 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.781976 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:43:53.782215 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:43:53.785825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42617
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:53.831478 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } attempt: 1
I20260927 00:43:53.832561 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.832198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43322)
0927 00:43:53.832366 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.832437 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:43:53.833273 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.833070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43330)
0927 00:43:53.833170 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.833218 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:43:53.833393 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
W20260927 00:43:53.895293 13804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:53.895565 13804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:53.895638 13804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:53.899587 13804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:53.899673 13804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:53.899717 13804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:53.899775 13804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:53.899812 13804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:53.901558 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } attempt: 2
I20260927 00:43:53.902379 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.902044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43344)
0927 00:43:53.902212 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.902276 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 00:43:53.903107 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.902898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43350)
0927 00:43:53.902999 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.903047 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 00:43:53.903278 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42617: connect: Connection refused (error 111)
I20260927 00:43:53.905527 13804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:53.906407 13804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:53.907299 13804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:53.915239 13810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:53.915306 13804 server_base.cc:1034] running on GCE node
W20260927 00:43:53.915256 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:53.915266 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:53.916020 13804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:53.917080 13804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:53.918248 13804 hybrid_clock.cc:648] HybridClock initialized: now 1790469833918215 us; error 44 us; skew 500 ppm
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:53.921146 13804 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:43:53.921934 13804 webserver.cc:460] Webserver started at http://127.0.98.61:40539/ using document root <none> and password file <none>
I20260927 00:43:53.922437 13804 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:53.922493 13804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:53.922649 13804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:53.924552 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data/instance:
uuid: "b717df4c9adc4ae5b47ba0e34e2148f7"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "6c0078eb1fe5f9e1ec91efe2ab41a397"
server_key_iv: "c790b23f16888d4c6441936bc49d6cef"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.925122 13804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal/instance:
uuid: "b717df4c9adc4ae5b47ba0e34e2148f7"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "6c0078eb1fe5f9e1ec91efe2ab41a397"
server_key_iv: "c790b23f16888d4c6441936bc49d6cef"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.928632 13804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:53.931010 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:53.932451 13804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:53.932618 13804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "b717df4c9adc4ae5b47ba0e34e2148f7"
format_stamp: "Formatted at 2026-09-27 00:43:53 on dist-test-slave-pkws"
server_key: "6c0078eb1fe5f9e1ec91efe2ab41a397"
server_key_iv: "c790b23f16888d4c6441936bc49d6cef"
server_key_version: "encryptionkey@0"
I20260927 00:43:53.932742 13804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:53.957137 13804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:53.960345 13804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:53.960551 13804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:53.967612 13804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42617
I20260927 00:43:53.967631 13871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42617 every 8 connection(s)
I20260927 00:43:53.968540 13804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:43:53.971164 13872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:53.971513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13804
I20260927 00:43:53.971644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-1/wal/instance
I20260927 00:43:53.971938 392 external_mini_cluster.cc:1428] Setting key 462a52c135cfd3cbc6bbc5c8816b89bd
I20260927 00:43:53.976161 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:53.990828 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.978468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.98.62:46181)
0927 00:43:53.978635 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:53.978641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:53.979432 (+ 791us) server_negotiation.cc:407] Connection header received
0927 00:43:53.980430 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:53.980436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:53.980510 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:53.980615 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:53.981949 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.982563 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.983880 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.984131 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.984255 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.987066 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:53.987069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:53.987072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:53.987096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:53.988556 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:53.989195 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:53.989198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:53.989199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:53.989274 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:53.989678 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:53.989680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:53.989681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:53.989851 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:53.989920 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:53.990589 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 00:43:53.990695 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:43:53.991688 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.978844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46181 (local address 127.0.0.1:34624)
0927 00:43:53.979292 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.979342 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:53.980205 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:53.980771 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:53.980777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:53.981346 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:53.981811 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.981824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.982732 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.982736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.983750 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.983759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.984288 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.984292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.984474 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.985100 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:53.985117 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:53.986889 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:53.988682 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:53.988690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:53.988707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:53.989095 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:53.989384 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:53.989387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:53.989389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:53.989574 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:53.989963 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:53.989972 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:53.990290 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 00:43:53.990514 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 00:43:53.990827 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260927 00:43:53.993130 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:53.994413 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } attempt: 3
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 27 00:43:53 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:43:54.000376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33493
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:54.003376 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.993614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43356)
0927 00:43:53.993733 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.993744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:53.993838 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:53.994366 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:53.994369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:53.994551 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:53.994641 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.994647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.995794 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.995800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.996479 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.996488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.997129 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.997132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.997217 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.997709 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:53.997720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:53.999117 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.001439 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.001444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.001447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.001735 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.002112 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.002117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.002120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.002226 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.002833 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.002839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.002942 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.003076 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.003198 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:43:54.003437 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.993724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.98.61:42617)
0927 00:43:53.994046 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:53.994055 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:53.994071 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:43:53.994101 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:53.994105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:53.994240 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:53.994380 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:53.994745 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.995690 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.996595 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.997019 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.997134 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.999261 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:53.999270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:53.999280 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:53.999300 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.001316 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.001883 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.001889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.001892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.001997 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.002395 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.002401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.002405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.002710 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.002856 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.003157 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.003278 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260927 00:43:54.004820 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.994811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43358)
0927 00:43:53.994926 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:53.994939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:53.995007 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:53.995409 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:53.995412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:53.995630 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:53.996139 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.996147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.997125 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.997129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.997852 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:53.997861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.998395 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.998400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:53.998528 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:53.998996 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:53.999011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.000487 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.003157 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.003163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.003168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.003442 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.003762 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.003768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.003772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.003863 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.004324 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.004327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.004423 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.004538 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.004678 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:43:54.005019 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:53.994872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.0.98.61:42617)
0927 00:43:53.995222 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:53.995229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:53.995239 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:43:53.995265 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:53.995266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:53.995296 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:53.995393 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:53.996253 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.996999 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.998027 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:53.998266 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:53.998376 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.000652 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.000656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.000659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.000682 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.003042 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.003569 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.003575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.003579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.003662 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.003976 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.003981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.003986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.004211 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.004317 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.004604 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.004735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260927 00:43:54.005699 13797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:54.005856 13872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:54.006430 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.006170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52298)
0927 00:43:54.006305 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.006365 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:43:54.006439 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.006216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52304)
0927 00:43:54.006320 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.006385 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:43:54.006614 13820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
I20260927 00:43:54.007184 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.006924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52316)
0927 00:43:54.007082 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.007126 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 00:43:54.007371 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
I20260927 00:43:54.007513 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.007296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52326)
0927 00:43:54.007403 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.007456 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 00:43:54.007681 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
I20260927 00:43:54.038518 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 1
I20260927 00:43:54.039260 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.038988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52330)
0927 00:43:54.039126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.039187 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 00:43:54.039974 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.039773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52344)
0927 00:43:54.039876 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.039921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:43:54.040129 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
I20260927 00:43:54.056964 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 1
I20260927 00:43:54.057803 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.057547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52354)
0927 00:43:54.057682 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.057737 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:43:54.058509 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.058301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52362)
0927 00:43:54.058406 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.058460 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:43:54.058634 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
W20260927 00:43:54.112185 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:54.112442 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:54.112525 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 00:43:54.115278 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 2
I20260927 00:43:54.116031 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.115744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52368)
0927 00:43:54.115893 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.115957 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 00:43:54.116688 13879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260927 00:43:54.116725 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.116513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52370)
0927 00:43:54.116628 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.116676 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:43:54.116757 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:54.116798 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:54.116859 13879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:54.116919 13879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 00:43:54.116909 13797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
I20260927 00:43:54.122085 13879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46181,127.0.98.61:42617,127.0.98.60:33493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:54.123015 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:54.123973 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:43:54.126778 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 2
I20260927 00:43:54.127437 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.127201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52380)
0927 00:43:54.127327 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.127376 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:43:54.128194 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.127966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52388)
0927 00:43:54.128086 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.128137 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:43:54.128361 13872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33493: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33493: connect: Connection refused (error 111)
W20260927 00:43:54.132186 13885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:54.132198 13888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:54.132205 13886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:54.132485 13879 server_base.cc:1034] running on GCE node
I20260927 00:43:54.133070 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:54.134091 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:54.135275 13879 hybrid_clock.cc:648] HybridClock initialized: now 1790469834135240 us; error 45 us; skew 500 ppm
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:54.138089 13879 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:43:54.138898 13879 webserver.cc:460] Webserver started at http://127.0.98.60:41663/ using document root <none> and password file <none>
I20260927 00:43:54.139433 13879 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:54.139523 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:54.139748 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:54.141757 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data/instance:
uuid: "d0b4a3a4484c4faa9250916b054085e8"
format_stamp: "Formatted at 2026-09-27 00:43:54 on dist-test-slave-pkws"
server_key: "d68fdbb0a2cd6e86d367daf8b9620826"
server_key_iv: "cf3e7ca016897d24aab0eb9e50c1c04a"
server_key_version: "encryptionkey@0"
I20260927 00:43:54.142328 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal/instance:
uuid: "d0b4a3a4484c4faa9250916b054085e8"
format_stamp: "Formatted at 2026-09-27 00:43:54 on dist-test-slave-pkws"
server_key: "d68fdbb0a2cd6e86d367daf8b9620826"
server_key_iv: "cf3e7ca016897d24aab0eb9e50c1c04a"
server_key_version: "encryptionkey@0"
I20260927 00:43:54.145782 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:43:54.147965 13895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:54.149142 13879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:43:54.149274 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "d0b4a3a4484c4faa9250916b054085e8"
format_stamp: "Formatted at 2026-09-27 00:43:54 on dist-test-slave-pkws"
server_key: "d68fdbb0a2cd6e86d367daf8b9620826"
server_key_iv: "cf3e7ca016897d24aab0eb9e50c1c04a"
server_key_version: "encryptionkey@0"
I20260927 00:43:54.149380 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:54.172036 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:54.174947 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:54.175158 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:54.182225 13879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33493
I20260927 00:43:54.182358 13947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33493 every 8 connection(s)
I20260927 00:43:54.183151 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:43:54.186026 13948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:54.187019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 13879
I20260927 00:43:54.187188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790469678562534-392-0/minicluster-data/master-2/wal/instance
I20260927 00:43:54.187477 392 external_mini_cluster.cc:1428] Setting key fca5f19a88e744acf94df0d29348220c
I20260927 00:43:54.190018 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:54.200873 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.191618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.0.98.62:46181)
0927 00:43:54.191762 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.191767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.192129 (+ 362us) server_negotiation.cc:407] Connection header received
0927 00:43:54.192255 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.192257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.192295 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.192401 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.193202 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.193924 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.194987 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.195207 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.195315 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.197599 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.197602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.197603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.197624 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.199142 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199592 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.199602 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.199604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.199670 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199942 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.199947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.199948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.200105 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.200207 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.200589 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.200704 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:43:54.201285 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.191487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46181 (local address 127.0.0.1:34638)
0927 00:43:54.191996 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.192043 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:54.192170 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:54.192442 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:54.192447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:54.192963 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:54.193087 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.193093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.194033 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.194036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.194866 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.194873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.195347 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.195351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.195461 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.195940 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:54.195954 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.197457 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.199252 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199262 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.199268 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.199483 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.199765 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.199770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.199851 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.200207 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.200214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.200360 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.200501 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.200688 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260927 00:43:54.201292 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.188695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.60:33493)
0927 00:43:54.189120 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.189135 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.189167 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:43:54.189837 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.189855 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.190159 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.190481 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.191240 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.192628 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.193786 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.194408 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.194556 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.197029 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.197035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.197045 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.197068 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.198580 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199003 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.199009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.199014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.199123 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.199415 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.199418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.199419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.199764 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.199970 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.200250 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.200500 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":153,"threads_started":1}
I20260927 00:43:54.202462 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:54.211194 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.202865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42617 (local address 127.0.0.1:43372)
0927 00:43:54.203085 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.203097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:54.203150 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:54.203673 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:54.203677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:54.203846 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:54.203974 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.203981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.204845 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.204849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.205572 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.205582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.206021 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.206025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.206108 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.206522 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:54.206535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.207813 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.209784 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.209787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.209789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.209985 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.210269 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.210272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.210273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.210363 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.210726 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.210729 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.210820 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.210924 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.211043 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260927 00:43:54.211228 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.202983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.98.61:42617)
0927 00:43:54.203127 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.203132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.203150 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:43:54.203216 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.203218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.203265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.203374 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.204091 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.204728 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.205682 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.205912 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.206017 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.207950 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.207952 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.207954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.207976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.209657 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.210084 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.210088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.210090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.210169 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.210460 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.210463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.210465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.210630 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.210735 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.210986 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.211096 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:43:54.211726 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.203607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.0.98.62:46181)
0927 00:43:54.203706 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.203710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.203729 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:43:54.203782 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.203784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.203814 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.203877 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.204461 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.205136 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.206050 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.206346 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.206451 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.208672 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.208675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.208678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.208699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.210144 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.210561 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.210564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.210566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.210637 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.210903 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.210907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.210909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.211065 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.211282 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.211395 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.211502 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:43:54.212201 13948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:43:54.216061 13797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 3
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:54.219518 13872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } attempt: 3
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:54.222183 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.212776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.61:42617)
0927 00:43:54.212880 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.212904 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.212935 (+ 31us) server_negotiation.cc:407] Connection header received
0927 00:43:54.213102 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.213105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.213142 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.213226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.213890 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.214323 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.215031 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.215236 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.215312 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.218393 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.218396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.218399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.218422 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.220059 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.220448 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.220452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.220453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.220533 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.220836 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.220841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.220844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.221082 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.221872 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.221943 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.222044 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:43:54.222872 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.212654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52410)
0927 00:43:54.212772 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.212785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:54.212871 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:54.213508 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:54.213514 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:54.213688 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:54.213768 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.213775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.215003 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.215006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.215672 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.215703 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.216144 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.216147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.216254 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.216832 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:54.216869 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.218642 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.221471 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.221473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.221475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.221691 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.221966 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.221968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.221970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.222048 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.222433 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.222436 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.222516 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.222608 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.222716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":20}
I20260927 00:43:54.223569 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.213217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.0.98.60:33493)
0927 00:43:54.213320 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.213323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.213335 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:43:54.213369 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.213371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.213398 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.213456 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.214439 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.214898 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.215810 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.216033 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.216147 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.218741 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.218744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.218746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.218792 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.221363 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.221792 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.221795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.221796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.221880 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.222139 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.222143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.222144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.222338 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.222441 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.222674 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.222759 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
Sep 27 00:43:54 dist-test-slave-pkws krb5kdc[13716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:43:54.226300 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.216495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52416)
0927 00:43:54.216639 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.216655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:54.216736 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:54.217287 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:54.217289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:54.217467 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:54.217534 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.217539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.218305 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.218310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.219027 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.219039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.219758 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.219765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.219870 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.220327 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:54.220343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.221804 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.224646 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.224649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.224652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.224915 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.225232 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.225235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.225236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.225315 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.225772 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.225775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.225868 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.225984 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.226122 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 00:43:54.226521 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:54.227510 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: Bootstrap starting.
I20260927 00:43:54.227629 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.216777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.0.98.60:33493)
0927 00:43:54.217098 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.217104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.217114 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:43:54.217140 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.217142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.217171 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.217272 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.217650 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.218200 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.219152 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.219574 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.219672 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.221981 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.221984 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.221987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.222011 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.224513 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.225037 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.225043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.225047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.225124 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.225439 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.225442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.225443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.225643 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.225775 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.226980 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.227396 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":144,"threads_started":1}
I20260927 00:43:54.229806 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.219860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33493 (local address 127.0.0.1:52424)
0927 00:43:54.220114 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:54.220129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:54.220200 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:54.221260 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:54.221262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:54.221438 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:54.221538 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.221546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.222678 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.222682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.223369 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:54.223378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.224037 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.224041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:54.224117 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.224550 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:54.224564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:54.225966 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:54.228136 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.228138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.228140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.228324 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.228703 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:54.228706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:54.228708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:54.228797 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:54.229299 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:54.229302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:54.229390 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:43:54.229470 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:43:54.229614 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 00:43:54.230441 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.220538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.0.98.60:33493)
0927 00:43:54.221012 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.221021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.221034 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:43:54.221065 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.221067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.221131 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.221647 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.221777 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.222543 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.223540 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.223903 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.224310 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.226103 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.226106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.226108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.226130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.228015 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.228456 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.228459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.228460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.228521 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.228967 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.228970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.228971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.229184 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.230089 (+ 905us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.230161 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.230270 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":113,"threads_started":1}
I20260927 00:43:54.230577 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: Bootstrap starting.
I20260927 00:43:54.233422 13797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:54.234261 13797 log.cc:824] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:54.236253 13797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: No bootstrap required, opened a new log
I20260927 00:43:54.236927 13948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:54.237749 13948 log.cc:824] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:54.238281 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: Bootstrap starting.
I20260927 00:43:54.240367 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.229198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.98.61:42617)
0927 00:43:54.229285 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.229289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.231430 (+ 2141us) server_negotiation.cc:407] Connection header received
0927 00:43:54.231590 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.231593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.231629 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.231674 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.232559 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.233233 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.234340 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.234947 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.235053 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.236824 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.236828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.236830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.236853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.238516 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.239050 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.239053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.239054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.239136 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.239465 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.239470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.239473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.239667 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.239761 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.240108 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.240210 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:43:54.240705 13872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:54.240676 13797 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.241117 13797 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:54.241220 13797 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef229d8c84af407badf53b7f94a834f2, State: Initialized, Role: FOLLOWER
I20260927 00:43:54.241577 13872 log.cc:824] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:54.241835 13797 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.243294 13797 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:54.243616 13872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: No bootstrap required, opened a new log
I20260927 00:43:54.244545 13948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: No bootstrap required, opened a new log
W20260927 00:43:54.245682 13750 tablet.cc:2369] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:54.247890 13872 raft_consensus.cc:348] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.248114 13872 raft_consensus.cc:374] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:54.248167 13872 raft_consensus.cc:729] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b717df4c9adc4ae5b47ba0e34e2148f7, State: Initialized, Role: FOLLOWER
I20260927 00:43:54.248860 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.230933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.98.60:33493)
0927 00:43:54.231013 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.231016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.238364 (+ 7348us) server_negotiation.cc:407] Connection header received
0927 00:43:54.238573 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.238577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.238618 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.238664 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.239463 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.240202 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.241261 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.241615 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.241693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.245059 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.245063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.245065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.245091 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.246818 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.247334 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.247338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.247339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.247435 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.247728 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.247732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.247733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.247924 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.248054 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.248607 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.248720 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 00:43:54.249253 13948 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.249472 13948 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:54.249526 13948 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b4a3a4484c4faa9250916b054085e8, State: Initialized, Role: FOLLOWER
I20260927 00:43:54.249598 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:54.228702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.0.98.62:46181)
0927 00:43:54.228800 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:54.228804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:54.229140 (+ 336us) server_negotiation.cc:407] Connection header received
0927 00:43:54.229311 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:54.229314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:54.229365 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:54.229416 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:54.230354 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.231002 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.232014 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:54.232278 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:54.232392 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:54.233817 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:54.233820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:54.233822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:54.233855 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:54.235515 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.242383 (+ 6868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.242388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.242390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.242478 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:54.248615 (+ 6137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:54.248618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:54.248620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:54.248813 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:54.248979 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:54.249324 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 00:43:54.249437 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 00:43:54.249939 13948 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.250458 13872 consensus_queue.cc:260] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.250993 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.251180 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.251222 13948 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:54.251683 13872 sys_catalog.cc:564] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:54.248274 13729 master_runner.cc:186] Error getting master registration for 127.0.98.61:42617: OK, instance_id { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" instance_seqno: 1790469833965772 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:43:54.252084 13729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:43:54.256033 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:54.256168 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 00:43:54.256467 13976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:54.256981 13976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:54.259443 13879 master_runner.cc:263] No leader master found from master d0b4a3a4484c4faa9250916b054085e8
I20260927 00:43:54.259506 13879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:54.260785 13804 master_runner.cc:263] No leader master found from master b717df4c9adc4ae5b47ba0e34e2148f7
I20260927 00:43:54.260847 13804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:43:54.265100 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.265234 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.265411 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.265518 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 00:43:54.268388 14002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:54.268464 14002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 00:43:54.426568 13901 tablet.cc:2369] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:54.454802 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:54.455025 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.456067 13965 leader_election.cc:288] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b717df4c9adc4ae5b47ba0e34e2148f7 (127.0.98.61:42617), d0b4a3a4484c4faa9250916b054085e8 (127.0.98.60:33493)
I20260927 00:43:54.456576 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef229d8c84af407badf53b7f94a834f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" is_pre_election: true
I20260927 00:43:54.456640 13922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef229d8c84af407badf53b7f94a834f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b4a3a4484c4faa9250916b054085e8" is_pre_election: true
I20260927 00:43:54.456928 13846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef229d8c84af407badf53b7f94a834f2 in term 0.
I20260927 00:43:54.456950 13922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef229d8c84af407badf53b7f94a834f2 in term 0.
I20260927 00:43:54.457367 13745 leader_election.cc:302] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0b4a3a4484c4faa9250916b054085e8, ef229d8c84af407badf53b7f94a834f2; no voters:
I20260927 00:43:54.457945 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:54.458033 13965 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:54.458076 13965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:54.459568 13965 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.459908 13965 leader_election.cc:288] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [CANDIDATE]: Term 1 election: Requested vote from peers b717df4c9adc4ae5b47ba0e34e2148f7 (127.0.98.61:42617), d0b4a3a4484c4faa9250916b054085e8 (127.0.98.60:33493)
I20260927 00:43:54.460251 13846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef229d8c84af407badf53b7f94a834f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7"
I20260927 00:43:54.460278 13922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef229d8c84af407badf53b7f94a834f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b4a3a4484c4faa9250916b054085e8"
I20260927 00:43:54.460362 13846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:54.460398 13922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:54.461549 13922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef229d8c84af407badf53b7f94a834f2 in term 1.
I20260927 00:43:54.461647 13846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef229d8c84af407badf53b7f94a834f2 in term 1.
I20260927 00:43:54.461863 13745 leader_election.cc:302] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0b4a3a4484c4faa9250916b054085e8, ef229d8c84af407badf53b7f94a834f2; no voters:
W20260927 00:43:54.462069 13825 tablet.cc:2369] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:54.462075 13965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:54.462320 13965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [term 1 LEADER]: Becoming Leader. State: Replica: ef229d8c84af407badf53b7f94a834f2, State: Running, Role: LEADER
I20260927 00:43:54.462644 13965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } }
I20260927 00:43:54.464488 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef229d8c84af407badf53b7f94a834f2. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.464582 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:54.464963 14006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:54.465793 14006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:54.469957 13922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [term 1 FOLLOWER]: Refusing update from remote peer ef229d8c84af407badf53b7f94a834f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:54.469972 13846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [term 1 FOLLOWER]: Refusing update from remote peer ef229d8c84af407badf53b7f94a834f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:54.470510 13965 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:54.470700 14004 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:54.474044 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef229d8c84af407badf53b7f94a834f2. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.474044 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef229d8c84af407badf53b7f94a834f2. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.474150 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.474150 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.476249 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.476372 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:54.476662 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.476754 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:54.476749 13978 sys_catalog.cc:455] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.476845 13978 sys_catalog.cc:458] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.477062 14009 mvcc.cc:205] Tried to move back new op lower bound from 7333764441983180800 to 7333764441961844736. Current Snapshot: MvccSnapshot[applied={T|T < 7333764441983180800}]
I20260927 00:43:54.477551 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef229d8c84af407badf53b7f94a834f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef229d8c84af407badf53b7f94a834f2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46181 } } peers { permanent_uuid: "b717df4c9adc4ae5b47ba0e34e2148f7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42617 } } peers { permanent_uuid: "d0b4a3a4484c4faa9250916b054085e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33493 } } }
I20260927 00:43:54.477649 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:43:54.477939 14006 catalog_manager.cc:1347] Generated new cluster ID: 772aa6e7cdef4708bfa9f4c06902019c
I20260927 00:43:54.477998 14006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:54.490036 14006 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:54.490860 14006 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:54.501995 14006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef229d8c84af407badf53b7f94a834f2: Generated new TSK 0
I20260927 00:43:54.502856 14006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:55.257536 13991 catalog_manager.cc:1259] Loaded cluster ID: 772aa6e7cdef4708bfa9f4c06902019c
I20260927 00:43:55.257618 13991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: loading cluster ID for follower catalog manager: success
I20260927 00:43:55.259174 13991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: acquiring CA information for follower catalog manager: success
I20260927 00:43:55.260290 13991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0b4a3a4484c4faa9250916b054085e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:55.269629 14002 catalog_manager.cc:1259] Loaded cluster ID: 772aa6e7cdef4708bfa9f4c06902019c
I20260927 00:43:55.269716 14002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: loading cluster ID for follower catalog manager: success
I20260927 00:43:55.271409 14002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: acquiring CA information for follower catalog manager: success
I20260927 00:43:55.272589 14002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b717df4c9adc4ae5b47ba0e34e2148f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:43:55.557672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13729
I20260927 00:43:55.563962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13804
I20260927 00:43:55.569563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 13879
2026-09-27T00:43:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3137 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:55 dist-test-slave-pkws krb5kdc[14022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:55 dist-test-slave-pkws krb5kdc[14022](info): set up 2 sockets
Sep 27 00:43:55 dist-test-slave-pkws krb5kdc[14022](info): commencing operation
krb5kdc: starting...
W20260927 00:43:56.605207 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:56 dist-test-slave-pkws krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:56.723311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41117
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:56.829686 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:56.829957 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:56.830025 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:56.833948 14035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:56.834021 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:56.834059 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:56.834116 14035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:56.834156 14035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:56.839150 14035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:56.840101 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:56.841120 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:56.849143 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:56.849156 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:56.849153 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:56.849539 14035 server_base.cc:1034] running on GCE node
I20260927 00:43:56.850031 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:56.851048 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:56.852205 14035 hybrid_clock.cc:648] HybridClock initialized: now 1790469836852192 us; error 27 us; skew 500 ppm
Sep 27 00:43:56 dist-test-slave-pkws krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:56.854984 14035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:56.855791 14035 webserver.cc:460] Webserver started at http://127.0.98.62:43727/ using document root <none> and password file <none>
I20260927 00:43:56.856320 14035 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:56.856380 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:56.856586 14035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:56.858626 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "bcb7c0b692254e4eb74d4c00382ae9f4"
format_stamp: "Formatted at 2026-09-27 00:43:56 on dist-test-slave-pkws"
server_key: "bace4c02efa052aa8d1632cb388786fd"
server_key_iv: "31f2b33ca862dfad2297a784e8411611"
server_key_version: "encryptionkey@0"
I20260927 00:43:56.859202 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcb7c0b692254e4eb74d4c00382ae9f4"
format_stamp: "Formatted at 2026-09-27 00:43:56 on dist-test-slave-pkws"
server_key: "bace4c02efa052aa8d1632cb388786fd"
server_key_iv: "31f2b33ca862dfad2297a784e8411611"
server_key_version: "encryptionkey@0"
I20260927 00:43:56.862708 14035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:43:56.865061 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:56.866248 14035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:43:56.866372 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "bcb7c0b692254e4eb74d4c00382ae9f4"
format_stamp: "Formatted at 2026-09-27 00:43:56 on dist-test-slave-pkws"
server_key: "bace4c02efa052aa8d1632cb388786fd"
server_key_iv: "31f2b33ca862dfad2297a784e8411611"
server_key_version: "encryptionkey@0"
I20260927 00:43:56.866474 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:56.876874 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:56.879639 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:56.879832 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:56.886560 14035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41117
I20260927 00:43:56.886590 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41117 every 8 connection(s)
I20260927 00:43:56.887470 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:56.889186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14035
I20260927 00:43:56.889319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:56.889582 392 external_mini_cluster.cc:1428] Setting key 90e46628c58a7880a73c18e112adacd7
I20260927 00:43:56.890470 14103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:56.897007 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: Bootstrap starting.
Sep 27 00:43:56 dist-test-slave-pkws krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:56.899648 14103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:56.900347 14103 log.cc:824] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:56.902139 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: No bootstrap required, opened a new log
I20260927 00:43:56.903823 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:56.890852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.0.98.62:41117)
0927 00:43:56.891246 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:56.891264 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:56.891304 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:43:56.892132 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:56.892153 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:56.892541 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:56.892926 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:56.893578 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:56.895029 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:56.895844 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:56.896503 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:56.896651 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:56.899288 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:56.899294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:56.899308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:56.899337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:56.901178 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:56.901635 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:56.901640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:56.901644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:56.901747 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:56.902024 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:56.902028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:56.902030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:56.902451 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:56.902646 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:56.902898 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 00:43:56.903163 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260927 00:43:56.905881 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } }
I20260927 00:43:56.906126 14103 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:56.906235 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb7c0b692254e4eb74d4c00382ae9f4, State: Initialized, Role: FOLLOWER
I20260927 00:43:56.906719 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } }
I20260927 00:43:56.906821 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:43:56.906881 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:56.906948 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:56.907042 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:56.907971 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } }
I20260927 00:43:56.908327 14103 leader_election.cc:302] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcb7c0b692254e4eb74d4c00382ae9f4; no voters:
I20260927 00:43:56.908659 14103 leader_election.cc:288] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:56.908799 14108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:56.909068 14108 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [term 1 LEADER]: Becoming Leader. State: Replica: bcb7c0b692254e4eb74d4c00382ae9f4, State: Running, Role: LEADER
I20260927 00:43:56.909469 14108 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } }
I20260927 00:43:56.909907 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:56.910989 14109 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } } }
I20260927 00:43:56.911116 14109 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:56.911274 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb7c0b692254e4eb74d4c00382ae9f4. Latest consensus state: current_term: 1 leader_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7c0b692254e4eb74d4c00382ae9f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41117 } } }
I20260927 00:43:56.911425 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4 [sys.catalog]: This master's current role is: LEADER
W20260927 00:43:56.914438 14123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:56.914539 14123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:56.914618 14118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:56.915314 14118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:56.920467 14118 catalog_manager.cc:1347] Generated new cluster ID: b0497fbd55804c3a841d2aebf10a2476
I20260927 00:43:56.920526 14118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:56.938797 14118 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:56.939954 14118 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:56.947650 14118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcb7c0b692254e4eb74d4c00382ae9f4: Generated new TSK 0
I20260927 00:43:56.948355 14118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:43:56.962381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14035
2026-09-27T00:43:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1390 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:43:57 dist-test-slave-pkws krb5kdc[14130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:43:57 dist-test-slave-pkws krb5kdc[14130](info): set up 2 sockets
Sep 27 00:43:57 dist-test-slave-pkws krb5kdc[14130](info): commencing operation
krb5kdc: starting...
W20260927 00:43:58.017284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:43:58.135325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:58.239807 14143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:58.240072 14143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:58.240144 14143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:58.244038 14143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:58.244118 14143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:58.244156 14143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:58.244216 14143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:58.244252 14143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:58.249189 14143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:58.250109 14143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:58.251030 14143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:58.259212 14148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.259274 14143 server_base.cc:1034] running on GCE node
W20260927 00:43:58.259196 14151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:58.259196 14149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.260051 14143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:58.261086 14143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:58.262259 14143 hybrid_clock.cc:648] HybridClock initialized: now 1790469838262240 us; error 30 us; skew 500 ppm
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:58.268316 14143 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:58.269276 14143 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260927 00:43:58.269829 14143 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:58.269894 14143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:58.270156 14143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:58.272113 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "c55b813a6b934b01b76ac15752ce10d1"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "50b3a5a1a19008a4e67094ba2ed10ee9"
server_key_iv: "79f090f77ff8cb0d4dea6422745ea9f7"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.272670 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "c55b813a6b934b01b76ac15752ce10d1"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "50b3a5a1a19008a4e67094ba2ed10ee9"
server_key_iv: "79f090f77ff8cb0d4dea6422745ea9f7"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.276275 14143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:43:58.278640 14158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.279807 14143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:43:58.279945 14143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "c55b813a6b934b01b76ac15752ce10d1"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "50b3a5a1a19008a4e67094ba2ed10ee9"
server_key_iv: "79f090f77ff8cb0d4dea6422745ea9f7"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.280051 14143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:58.303668 14143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:58.306602 14143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:58.306806 14143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:58.313956 14143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44203
I20260927 00:43:58.314229 14210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44203 every 8 connection(s)
I20260927 00:43:58.314896 14143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:58.317943 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:58.321308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14143
I20260927 00:43:58.321473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:43:58.321738 392 external_mini_cluster.cc:1428] Setting key 7a998f8b8bba228ecc5abe9004fb24c3
I20260927 00:43:58.323643 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap starting.
I20260927 00:43:58.326067 14211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:58.326792 14211 log.cc:824] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:58.328825 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: No bootstrap required, opened a new log
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:58.332868 14211 raft_consensus.cc:348] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:58.333168 14211 raft_consensus.cc:374] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:58.333231 14211 raft_consensus.cc:729] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Initialized, Role: FOLLOWER
I20260927 00:43:58.333698 14211 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:58.333861 14211 raft_consensus.cc:388] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:58.333969 14211 raft_consensus.cc:482] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:58.334065 14211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:58.335085 14211 raft_consensus.cc:504] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:58.335412 14211 leader_election.cc:302] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c55b813a6b934b01b76ac15752ce10d1; no voters:
I20260927 00:43:58.335727 14211 leader_election.cc:288] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:43:58.335841 14216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:58.336026 14216 raft_consensus.cc:686] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 LEADER]: Becoming Leader. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Running, Role: LEADER
I20260927 00:43:58.336166 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:58.323089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.0.98.62:44203)
0927 00:43:58.323483 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:58.323501 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:58.323544 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:43:58.324223 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:58.324245 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:58.324553 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:58.324843 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:58.325752 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.326789 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.327835 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.328339 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.328456 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:58.331258 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:58.331263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:58.331274 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:58.331300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:58.333065 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.333564 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.333570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.333574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.333704 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.334043 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.334047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.334049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.334505 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:58.334656 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:58.334941 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 00:43:58.335210 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":127,"threads_started":1}
I20260927 00:43:58.336367 14216 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:58.336860 14211 sys_catalog.cc:564] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:58.337527 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55b813a6b934b01b76ac15752ce10d1. Latest consensus state: current_term: 1 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:43:58.337653 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:58.337949 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:43:58.338079 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
W20260927 00:43:58.341732 14231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:43:58.341805 14231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:43:58.341907 14223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:58.342633 14223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:58.347645 14223 catalog_manager.cc:1347] Generated new cluster ID: 0257c21afe3d4c478c12070e6c29099a
I20260927 00:43:58.347754 14223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:58.354451 14223 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:43:58.355278 14223 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:58.360860 14223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Generated new TSK 0
I20260927 00:43:58.361481 14223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:43:58.390254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:58.499657 14237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:58.499871 14237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:58.499909 14237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:58.503789 14237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:58.503835 14237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:58.503888 14237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:58.508876 14237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:58.509814 14237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:58.510733 14237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:58.519784 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:58.519850 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:58.519802 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.519965 14237 server_base.cc:1034] running on GCE node
I20260927 00:43:58.520407 14237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:58.521032 14237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:58.522219 14237 hybrid_clock.cc:648] HybridClock initialized: now 1790469838522200 us; error 29 us; skew 500 ppm
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:58.524802 14237 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:58.525686 14237 webserver.cc:460] Webserver started at http://127.0.98.1:36733/ using document root <none> and password file <none>
I20260927 00:43:58.526149 14237 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:58.526211 14237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:58.526360 14237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:58.528162 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "422e52334e3b47cf803c90ac72d78ad3"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "eec9efeb288976b0cc750640c368e473"
server_key_iv: "12e631102be2bea06639a746baf1daa0"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.528643 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "422e52334e3b47cf803c90ac72d78ad3"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "eec9efeb288976b0cc750640c368e473"
server_key_iv: "12e631102be2bea06639a746baf1daa0"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.532141 14237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:43:58.534381 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.535584 14237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:43:58.535669 14237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "422e52334e3b47cf803c90ac72d78ad3"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "eec9efeb288976b0cc750640c368e473"
server_key_iv: "12e631102be2bea06639a746baf1daa0"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.535748 14237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:58.557134 14237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:58.560247 14237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:58.560390 14237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:58.560875 14237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:58.561661 14237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:58.561704 14237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.561744 14237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:58.561759 14237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.570360 14237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46633
I20260927 00:43:58.570390 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46633 every 8 connection(s)
I20260927 00:43:58.571174 14237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:58.576170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14237
I20260927 00:43:58.576329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:43:58.576619 392 external_mini_cluster.cc:1428] Setting key c4e3c5c102a35c9ae65f2c6ae942ce59
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:58.585443 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:58.572847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34647 (local address 127.0.98.62:44203)
0927 00:43:58.573024 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:58.573027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:58.574013 (+ 986us) server_negotiation.cc:407] Connection header received
0927 00:43:58.574884 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:58.574887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:58.574949 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:58.575027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:58.576151 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.576867 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.578033 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.578357 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.578458 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:58.581292 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:58.581296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:58.581298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:58.581323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:58.583190 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.583732 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.583735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.583737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.583833 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.584296 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.584299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.584300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.584468 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:58.584548 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:58.585189 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 00:43:58.585322 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:43:58.586555 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:58.573232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:34647)
0927 00:43:58.573841 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:58.573881 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:58.574680 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:58.575192 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:58.575198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:58.575557 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:58.575997 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:58.576010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.577023 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.577033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:58.577896 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:58.577906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.578524 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.578531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:58.578779 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:58.579430 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:58.579455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:58.581068 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:58.583342 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:58.583348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:58.583362 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:58.583624 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:58.583966 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:58.583971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:58.583973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:58.584183 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:58.584582 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:58.584590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:58.584931 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 00:43:58.585157 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 00:43:58.585524 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":101,"threads_started":1}
I20260927 00:43:58.587783 14366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:43:58.588019 14366 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:43:58.588567 14366 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:43:58.590250 14175 ts_manager.cc:194] Registered new tserver with Master: 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1:46633)
I20260927 00:43:58.591424 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:43:58.604272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:58.713613 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:58.713865 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:58.713948 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:58.718065 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:58.718116 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:58.718242 14371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:43:58.723304 14371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:58.724257 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:58.725221 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:58.734362 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.734488 14371 server_base.cc:1034] running on GCE node
W20260927 00:43:58.734449 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:58.734368 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.735280 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:58.735937 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:58.737144 14371 hybrid_clock.cc:648] HybridClock initialized: now 1790469838737118 us; error 48 us; skew 500 ppm
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:58.740206 14371 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:43:58.741093 14371 webserver.cc:460] Webserver started at http://127.0.98.2:39871/ using document root <none> and password file <none>
I20260927 00:43:58.741628 14371 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:58.741747 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:58.741966 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:58.743891 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "8befea83862149b4a8998d0b68bda268"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "5447bf2b822168dd1bd4235004fa14cc"
server_key_iv: "638c483cac8dd86b0ea8d4b43b0b07a9"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.744437 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8befea83862149b4a8998d0b68bda268"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "5447bf2b822168dd1bd4235004fa14cc"
server_key_iv: "638c483cac8dd86b0ea8d4b43b0b07a9"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.747964 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:43:58.750403 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.751603 14371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:43:58.751735 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "8befea83862149b4a8998d0b68bda268"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "5447bf2b822168dd1bd4235004fa14cc"
server_key_iv: "638c483cac8dd86b0ea8d4b43b0b07a9"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.751845 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:58.794603 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:58.797989 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:58.798190 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:58.798777 14371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:58.799594 14371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:58.799665 14371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.799737 14371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:58.799789 14371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.808511 14371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33007
I20260927 00:43:58.808560 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33007 every 8 connection(s)
I20260927 00:43:58.809523 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:43:58.810190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14371
I20260927 00:43:58.810336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:43:58.810652 392 external_mini_cluster.cc:1428] Setting key 7e6d9501a80b42f731fe097a2ed03ee6
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:43:58.824038 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:58.811510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60163 (local address 127.0.98.62:44203)
0927 00:43:58.811650 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:58.811655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:58.813124 (+ 1469us) server_negotiation.cc:407] Connection header received
0927 00:43:58.814019 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:58.814023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:58.814073 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:58.814159 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:58.815368 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.815989 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.817232 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.817471 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.817588 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:58.820316 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:58.820319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:58.820321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:58.820346 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:58.821785 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.822445 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.822448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.822449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.822523 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:58.822880 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:58.822885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:58.822886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:58.823069 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:58.823173 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:58.823769 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 00:43:58.823883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:43:58.825225 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:58.811968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:60163)
0927 00:43:58.812965 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:58.813017 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:58.813814 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:58.814354 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:58.814360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:58.814717 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:58.815219 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:58.815232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.816117 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.816120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:58.817090 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:58.817100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:58.817625 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:58.817629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:58.817809 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:58.818429 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:58.818444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:58.820157 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:58.821928 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:58.821934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:58.821948 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:58.822334 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:58.822629 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:58.822633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:58.822635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:58.822780 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:58.823186 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:58.823194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:58.823457 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 00:43:58.823689 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 00:43:58.824053 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":587,"threads_started":1}
I20260927 00:43:58.826460 14500 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:43:58.826705 14500 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:58.827181 14500 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:43:58.828202 14175 ts_manager.cc:194] Registered new tserver with Master: 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:43:58.828763 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:43:58.837750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:58.945210 14505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:58.945461 14505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:58.945533 14505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:58.949479 14505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:58.949549 14505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:58.949633 14505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:43:58.954806 14505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:58.955812 14505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:58.956758 14505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:58.965708 14511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.965826 14505 server_base.cc:1034] running on GCE node
W20260927 00:43:58.965673 14510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:58.965664 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:58.966344 14505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:58.966975 14505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:58.968158 14505 hybrid_clock.cc:648] HybridClock initialized: now 1790469838968143 us; error 25 us; skew 500 ppm
Sep 27 00:43:58 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:58.970876 14505 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:43:58.971676 14505 webserver.cc:460] Webserver started at http://127.0.98.3:41737/ using document root <none> and password file <none>
I20260927 00:43:58.972226 14505 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:58.972311 14505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:58.972584 14505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:43:58.974632 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "b775c329cbd241e8b953de25da0a9ef9"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "b0618973158a4b96677476dc98a24499"
server_key_iv: "437418258df21d42e2fc2ad803a86c9f"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.975188 14505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b775c329cbd241e8b953de25da0a9ef9"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "b0618973158a4b96677476dc98a24499"
server_key_iv: "437418258df21d42e2fc2ad803a86c9f"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.978818 14505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:43:58.981114 14520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:58.982333 14505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:58.982475 14505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b775c329cbd241e8b953de25da0a9ef9"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "b0618973158a4b96677476dc98a24499"
server_key_iv: "437418258df21d42e2fc2ad803a86c9f"
server_key_version: "encryptionkey@0"
I20260927 00:43:58.982591 14505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:59.011013 14505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:59.014071 14505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:59.014221 14505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:59.014717 14505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:59.015447 14505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:43:59.015492 14505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:59.015529 14505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:43:59.015545 14505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:59.024610 14505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46259
I20260927 00:43:59.024724 14633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46259 every 8 connection(s)
I20260927 00:43:59.025498 14505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:43:59.033377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14505
I20260927 00:43:59.033515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:43:59.033784 392 external_mini_cluster.cc:1428] Setting key 9a4ba3593fa061bc4d5e5cf6b2886eb3
Sep 27 00:43:59 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:59.039395 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.027259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60369 (local address 127.0.98.62:44203)
0927 00:43:59.027444 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.027449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.028631 (+ 1182us) server_negotiation.cc:407] Connection header received
0927 00:43:59.029552 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.029554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.029599 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.029689 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:59.030789 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.031318 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.032411 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.032634 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.032725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.035611 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:59.035614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:59.035617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:59.035641 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:59.037341 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.037847 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.037852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.037854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.037939 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.038311 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.038316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.038318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.038506 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:59.038612 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.039074 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.039223 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 00:43:59.040102 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.027604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:60369)
0927 00:43:59.028476 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.028517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.029358 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.029832 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.029838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.030206 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:59.030651 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.030666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.031431 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.031434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.032293 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.032302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.032767 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.032770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.032924 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.033524 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:59.033549 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:59.035462 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:59.037455 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.037460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.037473 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.037726 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.038044 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.038047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.038049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.038202 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.038609 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:59.038614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:59.038842 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.039028 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.039358 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":542,"threads_started":1}
I20260927 00:43:59.041239 14634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:43:59.041440 14634 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:59.041864 14634 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:43:59.042914 14175 ts_manager.cc:194] Registered new tserver with Master: b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259)
I20260927 00:43:59.043466 14175 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60369
I20260927 00:43:59.047680 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:43:59.056721 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.049693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.62:44203)
0927 00:43:59.049815 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.049818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.049952 (+ 134us) server_negotiation.cc:407] Connection header received
0927 00:43:59.050065 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.050067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.050107 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.050171 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:59.050814 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.051322 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.052074 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.052292 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.052363 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.053646 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:59.053649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:59.053651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:59.053671 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:59.055132 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.055510 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.055512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.055513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.055584 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.055850 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.055853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.055854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.056102 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:59.056271 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.056519 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.056611 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:43:59.059535 14175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:43:59.063913 14175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:43:59.081913 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.073901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46259 (local address 127.0.0.1:49276)
0927 00:43:59.074247 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.074300 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.074431 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.075956 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.075961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.075984 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:59.076101 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.076108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.077464 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.077468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.078956 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.078968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.081507 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.081510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.081593 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.081615 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.081663 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.081741 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260927 00:43:59.082345 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.073991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.0.98.2:33007)
0927 00:43:59.074553 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.074565 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.074881 (+ 316us) server_negotiation.cc:407] Connection header received
0927 00:43:59.074967 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.074973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.075253 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.075427 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:59.075911 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.077304 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.080076 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.081106 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.081218 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.081354 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.082085 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.082198 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":109,"threads_started":1}
I20260927 00:43:59.081811 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.073955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.0.1:45464)
0927 00:43:59.074692 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.074704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.074787 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.075618 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.075624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.075645 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:59.075773 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.075780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.077454 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.077459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.079952 (+ 2493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.079965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.081232 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.081238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.081381 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.081425 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.081536 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.081667 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1620,"thread_start_us":77,"threads_started":1}
I20260927 00:43:59.082655 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.075011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.98.3:46259)
0927 00:43:59.075514 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.075526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.075546 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:43:59.075616 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.075621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.075813 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.075961 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:59.076227 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.077316 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.079081 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.080002 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.081876 (+ 1874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.082108 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.082161 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.082249 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":300,"threads_started":1}
I20260927 00:43:59.083927 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.074595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46633 (local address 127.0.0.1:59806)
0927 00:43:59.077578 (+ 2983us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.077590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.077683 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.078148 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.078154 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.078191 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:59.078295 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.078301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.080121 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.080125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.081351 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.081361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.082417 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.082423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.083296 (+ 873us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.083321 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.083394 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.083805 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2928,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260927 00:43:59.084141 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.074988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.0.98.1:46633)
0927 00:43:59.075437 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.075454 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.077754 (+ 2300us) server_negotiation.cc:407] Connection header received
0927 00:43:59.077833 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.077838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.078006 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.078153 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:59.078407 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.079208 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.081449 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.082310 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.082423 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.082646 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.083476 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.083567 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":68,"threads_started":1}
I20260927 00:43:59.085551 14434 tablet_service.cc:1467] Processing CreateTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 (DEFAULT_TABLE table=test-table [id=44807019bf364b3c8b209ebea8783704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:59.086674 14434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a356c7fedd2944a9bff4afb8295f0d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:59.086637 14300 tablet_service.cc:1467] Processing CreateTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 (DEFAULT_TABLE table=test-table [id=44807019bf364b3c8b209ebea8783704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:59.087620 14300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a356c7fedd2944a9bff4afb8295f0d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:59.087469 14568 tablet_service.cc:1467] Processing CreateTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 (DEFAULT_TABLE table=test-table [id=44807019bf364b3c8b209ebea8783704]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:43:59.088467 14568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a356c7fedd2944a9bff4afb8295f0d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:43:59.094187 14651 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap starting.
I20260927 00:43:59.095516 14652 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Bootstrap starting.
I20260927 00:43:59.096133 14653 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Bootstrap starting.
I20260927 00:43:59.097263 14651 tablet_bootstrap.cc:654] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:59.098001 14652 tablet_bootstrap.cc:654] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:59.098080 14653 tablet_bootstrap.cc:654] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Neither blocks nor log segments found. Creating new log.
I20260927 00:43:59.098361 14651 log.cc:824] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:59.098585 14652 log.cc:824] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:59.099261 14653 log.cc:824] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:59.100422 14651 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: No bootstrap required, opened a new log
I20260927 00:43:59.100422 14652 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: No bootstrap required, opened a new log
I20260927 00:43:59.100605 14652 ts_tablet_manager.cc:1397] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:43:59.100605 14651 ts_tablet_manager.cc:1397] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:43:59.101094 14653 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: No bootstrap required, opened a new log
I20260927 00:43:59.101274 14653 ts_tablet_manager.cc:1397] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:43:59.104684 14652 raft_consensus.cc:348] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.105111 14652 raft_consensus.cc:374] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:59.104997 14651 raft_consensus.cc:348] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.105226 14651 raft_consensus.cc:374] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:59.105207 14652 raft_consensus.cc:729] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8befea83862149b4a8998d0b68bda268, State: Initialized, Role: FOLLOWER
I20260927 00:43:59.105278 14651 raft_consensus.cc:729] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Initialized, Role: FOLLOWER
I20260927 00:43:59.105762 14651 consensus_queue.cc:260] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.105777 14652 consensus_queue.cc:260] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.105867 14653 raft_consensus.cc:348] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.106142 14653 raft_consensus.cc:374] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:43:59.106204 14653 raft_consensus.cc:729] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b775c329cbd241e8b953de25da0a9ef9, State: Initialized, Role: FOLLOWER
I20260927 00:43:59.106616 14366 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
I20260927 00:43:59.106657 14653 consensus_queue.cc:260] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.106781 14651 ts_tablet_manager.cc:1428] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:43:59.107254 14500 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
I20260927 00:43:59.107288 14634 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
I20260927 00:43:59.108301 14652 ts_tablet_manager.cc:1428] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 00:43:59.108812 14653 ts_tablet_manager.cc:1428] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 00:43:59.276433 14635 tablet.cc:2369] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:59.310626 14501 tablet.cc:2369] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:43:59.322417 14367 tablet.cc:2369] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:43:59.386829 14657 raft_consensus.cc:482] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:43:59.387064 14657 raft_consensus.cc:504] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.388234 14657 leader_election.cc:288] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:43:59.392195 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.388600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60527 (local address 127.0.98.2:33007)
0927 00:43:59.388724 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.388731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.388993 (+ 262us) server_negotiation.cc:407] Connection header received
0927 00:43:59.389157 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.389166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.389211 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.389297 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:59.389610 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.390110 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391180 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.391631 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.391816 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.392009 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.392078 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:43:59.392218 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.388519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.98.1:60527)
0927 00:43:59.388857 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.388869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.389047 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.389318 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.389321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.389354 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:59.389498 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.389503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.390217 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.390224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.391053 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.391061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.391759 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.391859 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.391894 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.391993 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.392061 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":4,"thread_start_us":123,"threads_started":1}
I20260927 00:43:59.392386 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.388561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35495 (local address 127.0.98.3:46259)
0927 00:43:59.388699 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.388703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.389098 (+ 395us) server_negotiation.cc:407] Connection header received
0927 00:43:59.389241 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.389244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.389303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.389406 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:43:59.389683 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.390207 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391300 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.391726 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391789 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.391837 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.392128 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.392198 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:43:59.392717 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.388524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46259 (local address 127.0.98.1:35495)
0927 00:43:59.389009 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.389022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.389138 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.389419 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.389422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.389438 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:43:59.389571 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.389577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.390330 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.390334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.391117 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.391125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.391914 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.391916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.392002 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.392016 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.392101 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.392179 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":22,"thread_start_us":143,"threads_started":1}
I20260927 00:43:59.393069 14454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a356c7fedd2944a9bff4afb8295f0d89" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8befea83862149b4a8998d0b68bda268" is_pre_election: true
I20260927 00:43:59.393432 14454 raft_consensus.cc:2454] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 0.
I20260927 00:43:59.393435 14588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a356c7fedd2944a9bff4afb8295f0d89" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775c329cbd241e8b953de25da0a9ef9" is_pre_election: true
I20260927 00:43:59.393687 14588 raft_consensus.cc:2454] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 0.
I20260927 00:43:59.393952 14256 leader_election.cc:302] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3, 8befea83862149b4a8998d0b68bda268; no voters:
I20260927 00:43:59.394239 14657 raft_consensus.cc:2781] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:43:59.394316 14657 raft_consensus.cc:482] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:43:59.394347 14657 raft_consensus.cc:3037] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:59.395375 14657 raft_consensus.cc:504] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.395737 14657 leader_election.cc:288] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 election: Requested vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:43:59.396101 14454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a356c7fedd2944a9bff4afb8295f0d89" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8befea83862149b4a8998d0b68bda268"
I20260927 00:43:59.396101 14588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a356c7fedd2944a9bff4afb8295f0d89" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775c329cbd241e8b953de25da0a9ef9"
I20260927 00:43:59.396222 14454 raft_consensus.cc:3037] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:59.396246 14588 raft_consensus.cc:3037] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:43:59.397549 14588 raft_consensus.cc:2454] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 1.
I20260927 00:43:59.397569 14454 raft_consensus.cc:2454] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 1.
I20260927 00:43:59.397858 14255 leader_election.cc:302] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3, b775c329cbd241e8b953de25da0a9ef9; no voters:
I20260927 00:43:59.398066 14657 raft_consensus.cc:2781] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:43:59.398281 14657 raft_consensus.cc:686] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Running, Role: LEADER
I20260927 00:43:59.398624 14657 consensus_queue.cc:237] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.401702 14175 catalog_manager.cc:5495] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "422e52334e3b47cf803c90ac72d78ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } health_report { overall_health: HEALTHY } } }
I20260927 00:43:59.440409 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.436840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.0.98.1:46633)
0927 00:43:59.436995 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.437002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.437035 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:43:59.437131 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.437133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.437202 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.437290 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:43:59.437514 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.438111 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.438932 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.439224 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.439298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.439597 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:43:59.440011 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:43:59.440102 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.440189 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.440254 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 00:43:59.446091 14588 raft_consensus.cc:1261] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Refusing update from remote peer 422e52334e3b47cf803c90ac72d78ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:59.446090 14454 raft_consensus.cc:1261] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Refusing update from remote peer 422e52334e3b47cf803c90ac72d78ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:43:59.446789 14662 consensus_queue.cc:1035] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:59.446995 14657 consensus_queue.cc:1035] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:43:59.462742 14175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:59.462930 14175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:43:59.465445 14175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Sending DeleteTablet for 3 replicas of tablet a356c7fedd2944a9bff4afb8295f0d89
I20260927 00:43:59.466246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14237
I20260927 00:43:59.466216 14568 tablet_service.cc:1514] Processing DeleteTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49276
I20260927 00:43:59.466691 14434 tablet_service.cc:1514] Processing DeleteTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260927 00:43:59.467162 14675 tablet_replica.cc:330] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: stopping tablet replica
I20260927 00:43:59.467514 14675 raft_consensus.cc:2229] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:59.467829 14675 raft_consensus.cc:2258] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:59.468600 14674 tablet_replica.cc:330] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: stopping tablet replica
I20260927 00:43:59.468849 14674 raft_consensus.cc:2229] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:59.469134 14674 raft_consensus.cc:2258] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:59.470371 14674 ts_tablet_manager.cc:1905] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:43:59.472468 14161 connection.cc:535] client connection to 127.0.98.1:46633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 00:43:59.472623 14161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:43:59.472782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14371
I20260927 00:43:59.474797 14674 ts_tablet_manager.cc:1918] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:59.474850 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.474511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46633 (local address 127.0.0.1:59818)
0927 00:43:59.474665 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.474746 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:43:59.474924 14674 log.cc:1196] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/a356c7fedd2944a9bff4afb8295f0d89
W20260927 00:43:59.475023 14161 catalog_manager.cc:4584] TS 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1:46633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a356c7fedd2944a9bff4afb8295f0d89: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46633: connect: Connection refused (error 111)
I20260927 00:43:59.475279 14674 ts_tablet_manager.cc:1939] T a356c7fedd2944a9bff4afb8295f0d89 P b775c329cbd241e8b953de25da0a9ef9: Deleting consensus metadata
I20260927 00:43:59.476348 14161 catalog_manager.cc:4841] TS b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): tablet a356c7fedd2944a9bff4afb8295f0d89 (table test-table [id=44807019bf364b3c8b209ebea8783704]) successfully deleted
I20260927 00:43:59.478761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14505
I20260927 00:43:59.480727 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.478990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.0.1:45480)
0927 00:43:59.480551 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.480622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1455}
I20260927 00:43:59.484880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14143
I20260927 00:43:59.491508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--webserver_interface=127.0.98.62
--webserver_port=43883
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:43:59.601676 14677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:59.601912 14677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:59.601950 14677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:59.605850 14677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:43:59.605898 14677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:59.605918 14677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:59.605935 14677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:43:59.605952 14677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:43:59.610884 14677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:59.611796 14677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:59.612730 14677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:59.625461 14683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:59.625566 14684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:59.625461 14686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:59.625752 14677 server_base.cc:1034] running on GCE node
I20260927 00:43:59.626329 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:59.627389 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:59.628559 14677 hybrid_clock.cc:648] HybridClock initialized: now 1790469839628544 us; error 26 us; skew 500 ppm
Sep 27 00:43:59 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:59.631407 14677 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:43:59.632303 14677 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260927 00:43:59.632881 14677 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:59.633036 14677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:59.636662 14677 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:43:59.638798 14693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:59.640019 14677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:43:59.640172 14677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "c55b813a6b934b01b76ac15752ce10d1"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "50b3a5a1a19008a4e67094ba2ed10ee9"
server_key_iv: "79f090f77ff8cb0d4dea6422745ea9f7"
server_key_version: "encryptionkey@0"
I20260927 00:43:59.640929 14677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:59.677151 14677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:59.680176 14677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:59.680387 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:59.687403 14677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44203
I20260927 00:43:59.687736 14745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44203 every 8 connection(s)
I20260927 00:43:59.688467 14677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:43:59.692756 14746 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:43:59.694376 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap starting.
I20260927 00:43:59.697698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14677
I20260927 00:43:59.698127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46633
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=36733
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:59.711076 14746 log.cc:824] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:43:59.719978 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:43:59.720382 14746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap complete.
I20260927 00:43:59.724354 14746 raft_consensus.cc:348] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:59.724709 14746 raft_consensus.cc:729] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Initialized, Role: FOLLOWER
I20260927 00:43:59.725244 14746 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:59.725358 14746 raft_consensus.cc:388] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:43:59.725458 14746 raft_consensus.cc:482] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:43:59.725571 14746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:43:59.727396 14746 raft_consensus.cc:504] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:59.727716 14746 leader_election.cc:302] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c55b813a6b934b01b76ac15752ce10d1; no voters:
I20260927 00:43:59.728032 14746 leader_election.cc:288] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:43:59.728133 14750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:43:59.728340 14750 raft_consensus.cc:686] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 LEADER]: Becoming Leader. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Running, Role: LEADER
I20260927 00:43:59.728652 14750 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:43:59.729264 14746 sys_catalog.cc:564] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:43:59.729570 14750 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55b813a6b934b01b76ac15752ce10d1. Latest consensus state: current_term: 2 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:43:59.729687 14750 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:59.730058 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:43:59.730234 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:43:59.731228 14754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:43:59.736147 14754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:43:59.736734 14754 catalog_manager.cc:1259] Loaded cluster ID: 0257c21afe3d4c478c12070e6c29099a
I20260927 00:43:59.736794 14754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:43:59.738153 14754 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 00:43:59.742686 14754 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 00:43:59.747802 14754 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:43:59.748473 14754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Loaded TSK: 0
I20260927 00:43:59.749661 14754 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:43:59.812791 14748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:43:59.813108 14748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:43:59.813190 14748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:43:59.817216 14748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:43:59.817271 14748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:43:59.817358 14748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:43:59.822559 14748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36733
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:43:59.823495 14748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:43:59.824728 14748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:43:59.837037 14773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:43:59.837049 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:59.837260 14748 server_base.cc:1034] running on GCE node
W20260927 00:43:59.837428 14774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:43:59.837921 14748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:43:59.838649 14748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:43:59.839869 14748 hybrid_clock.cc:648] HybridClock initialized: now 1790469839839837 us; error 32 us; skew 500 ppm
Sep 27 00:43:59 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:43:59.843176 14748 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:43:59.844071 14748 webserver.cc:460] Webserver started at http://127.0.98.1:36733/ using document root <none> and password file <none>
I20260927 00:43:59.844696 14748 fs_manager.cc:359] Metadata directory not provided
I20260927 00:43:59.844772 14748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:43:59.848842 14748 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:43:59.851109 14783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:43:59.852448 14748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:43:59.852591 14748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "422e52334e3b47cf803c90ac72d78ad3"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "eec9efeb288976b0cc750640c368e473"
server_key_iv: "12e631102be2bea06639a746baf1daa0"
server_key_version: "encryptionkey@0"
I20260927 00:43:59.853247 14748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:43:59.866788 14748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:43:59.869940 14748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:43:59.870159 14748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:43:59.870714 14748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:43:59.871826 14790 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:43:59.876667 14748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:43:59.876748 14748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 00:43:59.876827 14748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:43:59.878479 14748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:43:59.878547 14748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:43:59.878808 14790 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap starting.
I20260927 00:43:59.888981 14748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46633
I20260927 00:43:59.889052 14897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46633 every 8 connection(s)
I20260927 00:43:59.889912 14748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:43:59.894372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14748
I20260927 00:43:59.894863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33007
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=39871
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:43:59.898120 14790 log.cc:824] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:43:59 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:43:59.906397 14790 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:43:59.906809 14790 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap complete.
I20260927 00:43:59.908088 14790 ts_tablet_manager.cc:1397] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.012s
I20260927 00:43:59.909780 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.891995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:56673)
0927 00:43:59.892564 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.892620 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.893854 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.895347 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.895356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.895926 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:59.896623 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.896644 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.898651 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.898657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.899664 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.899673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.900779 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.900783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.901002 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.901687 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:59.901706 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:59.903606 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:59.906384 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.906390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.906402 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.906760 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.907188 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.907194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.907199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.907377 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.908020 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:59.908027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:59.908321 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.908527 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.908833 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":136,"threads_started":1}
I20260927 00:43:59.910101 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.892085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56673 (local address 127.0.98.62:44203)
0927 00:43:59.893102 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.893117 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.893161 (+ 44us) server_negotiation.cc:407] Connection header received
0927 00:43:59.894403 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.894419 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.894854 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.895217 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:59.896823 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.898480 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.899794 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.900614 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.900815 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.903822 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:59.903830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:59.903849 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:59.903888 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:59.906243 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.906918 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.906925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.906930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.907073 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.907480 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.907484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.907486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.907897 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:59.908025 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.908661 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.908968 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":126,"threads_started":1}
I20260927 00:43:59.910825 14790 raft_consensus.cc:348] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.911253 14790 raft_consensus.cc:729] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Initialized, Role: FOLLOWER
I20260927 00:43:59.911731 14790 consensus_queue.cc:260] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:43:59.912592 14898 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:43:59.912693 14790 ts_tablet_manager.cc:1428] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 00:43:59.912811 14898 heartbeater.cc:464] Registering TS with master...
I20260927 00:43:59.913277 14898 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:43:59.915632 14710 ts_manager.cc:194] Registered new tserver with Master: 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1:46633)
I20260927 00:43:59.919485 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56673
I20260927 00:43:59.922024 14898 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
Sep 27 00:43:59 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:43:59.928761 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.918761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46633 (local address 127.0.0.1:59832)
0927 00:43:59.919423 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 00:43:59.919455 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 00:43:59.919607 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:43:59.920038 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:43:59.920044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:43:59.920509 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:43:59.920608 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.920614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.921615 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.921621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.922300 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:43:59.922310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.923058 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.923074 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:43:59.923177 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.923723 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:43:59.923734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:43:59.925199 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:43:59.927172 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.927179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.927183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.927401 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.927699 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:43:59.927701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:43:59.927702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:43:59.927773 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:43:59.928294 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:43:59.928298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:43:59.928390 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:43:59.928514 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 00:43:59.928640 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":111,"threads_started":1}
I20260927 00:43:59.929137 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:43:59.918831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.0.98.1:46633)
0927 00:43:59.919182 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 00:43:59.919193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:43:59.919547 (+ 354us) server_negotiation.cc:407] Connection header received
0927 00:43:59.919736 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:43:59.919748 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:43:59.919898 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:43:59.920037 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:43:59.920726 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.921497 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.922453 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:43:59.922929 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:43:59.923038 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:43:59.925335 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:43:59.925345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:43:59.925356 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:43:59.925381 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:43:59.927055 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.927506 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.927511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.927513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.927607 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:43:59.927857 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:43:59.927859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:43:59.927861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:43:59.928191 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:43:59.928315 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:43:59.928613 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:43:59.928769 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":90,"threads_started":1}
I20260927 00:43:59.930099 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260927 00:43:59.931027 14907 tablet_replica.cc:330] stopping tablet replica
I20260927 00:43:59.931300 14907 raft_consensus.cc:2229] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:43:59.931566 14907 raft_consensus.cc:2258] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:43:59.932477 14907 ts_tablet_manager.cc:1905] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:43:59.935750 14907 ts_tablet_manager.cc:1918] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:43:59.935863 14907 log.cc:1196] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/a356c7fedd2944a9bff4afb8295f0d89
I20260927 00:43:59.936132 14907 ts_tablet_manager.cc:1939] T a356c7fedd2944a9bff4afb8295f0d89 P 422e52334e3b47cf803c90ac72d78ad3: Deleting consensus metadata
I20260927 00:43:59.937774 14696 catalog_manager.cc:4841] TS 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1:46633): tablet a356c7fedd2944a9bff4afb8295f0d89 (table test-table [id=44807019bf364b3c8b209ebea8783704]) successfully deleted
W20260927 00:44:00.015758 14902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:00.015986 14902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:00.016027 14902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:00.020103 14902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:00.020155 14902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:00.020246 14902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:00.025457 14902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39871
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:00.026389 14902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:00.027343 14902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:00.036880 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.036880 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.036882 14912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:00.036944 14902 server_base.cc:1034] running on GCE node
I20260927 00:44:00.037509 14902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:00.038126 14902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:00.039315 14902 hybrid_clock.cc:648] HybridClock initialized: now 1790469840039282 us; error 53 us; skew 500 ppm
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:00.042649 14902 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:00.043807 14902 webserver.cc:460] Webserver started at http://127.0.98.2:39871/ using document root <none> and password file <none>
I20260927 00:44:00.044466 14902 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:00.044538 14902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:00.048619 14902 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:00.050889 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:00.052210 14902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:44:00.052343 14902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "8befea83862149b4a8998d0b68bda268"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "5447bf2b822168dd1bd4235004fa14cc"
server_key_iv: "638c483cac8dd86b0ea8d4b43b0b07a9"
server_key_version: "encryptionkey@0"
I20260927 00:44:00.052843 14902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:00.103880 14902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:00.107430 14902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:00.107643 14902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:00.108265 14902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:00.109392 14929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:44:00.113468 14902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:44:00.113543 14902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 00:44:00.113667 14902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:44:00.115341 14902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:44:00.115406 14902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:44:00.115650 14929 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Bootstrap starting.
I20260927 00:44:00.121726 14929 log.cc:824] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:00.125480 14902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33007
I20260927 00:44:00.125501 15037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33007 every 8 connection(s)
I20260927 00:44:00.126293 14902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:00.129709 14929 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:00.130187 14929 tablet_bootstrap.cc:492] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Bootstrap complete.
I20260927 00:44:00.130519 14929 ts_tablet_manager.cc:1397] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260927 00:44:00.133251 14929 raft_consensus.cc:348] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:44:00.133785 14929 raft_consensus.cc:729] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8befea83862149b4a8998d0b68bda268, State: Initialized, Role: FOLLOWER
I20260927 00:44:00.134280 14929 consensus_queue.cc:260] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } }
I20260927 00:44:00.135313 14929 ts_tablet_manager.cc:1428] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:44:00.135641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 14902
I20260927 00:44:00.135985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46259
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=41737
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:00.142658 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.128076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47271 (local address 127.0.98.62:44203)
0927 00:44:00.128231 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.128236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.128816 (+ 580us) server_negotiation.cc:407] Connection header received
0927 00:44:00.129968 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.129971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.130075 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.130174 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:00.131582 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.132129 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.133788 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.134126 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.134225 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.138952 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:00.138955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:00.138957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:00.138982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:00.140485 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.141134 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.141138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.141140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.141224 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.141568 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.141571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.141573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.141762 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:00.141872 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.142358 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.142489 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:44:00.143383 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.128177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:47271)
0927 00:44:00.128646 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.128695 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.129728 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.130343 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.130351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.130824 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:00.131425 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.131442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.132277 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.132281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.133588 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.133597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.134250 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.134254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.134456 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.136987 (+ 2531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:00.137014 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:00.138776 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:00.140625 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.140631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.140644 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.141029 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.141319 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.141324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.141326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.141466 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.141869 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:00.141875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:00.142119 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.142302 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.142603 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260927 00:44:00.144562 15038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:44:00.144810 15038 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:00.145364 15038 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:44:00.146924 14710 ts_manager.cc:194] Registered new tserver with Master: 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:44:00.148079 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47271
I20260927 00:44:00.150295 15038 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:44:00.157584 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.147783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.0.1:45482)
0927 00:44:00.147945 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.147959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.148027 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.148884 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.148907 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.149101 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:00.149195 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.149202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.150680 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.150686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.151479 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.151488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.152090 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.152105 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.152196 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.152680 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:00.152694 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:00.153925 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:00.155848 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.155853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.155855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.156053 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.156436 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.156438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.156439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.156514 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.157009 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:00.157014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:00.157143 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.157293 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.157435 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 00:44:00.157702 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.147892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.0.98.2:33007)
0927 00:44:00.148237 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.148248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.148268 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:44:00.148479 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.148541 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.148748 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.148877 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:00.149316 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.150523 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.151612 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.151979 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.152102 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.154058 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:00.154067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:00.154091 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:00.154111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:00.155731 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.156218 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.156223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.156224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.156299 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.156617 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.156620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.156622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.156851 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:00.157035 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.157369 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.157536 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":259,"thread_start_us":87,"threads_started":1}
I20260927 00:44:00.158586 14971 tablet_service.cc:1514] Processing DeleteTablet for tablet a356c7fedd2944a9bff4afb8295f0d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:43:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45482
I20260927 00:44:00.159432 15044 tablet_replica.cc:330] stopping tablet replica
I20260927 00:44:00.159718 15044 raft_consensus.cc:2229] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:00.159955 15044 raft_consensus.cc:2258] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:00.160846 15044 ts_tablet_manager.cc:1905] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:00.164273 15044 ts_tablet_manager.cc:1918] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:00.164350 15044 log.cc:1196] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/a356c7fedd2944a9bff4afb8295f0d89
I20260927 00:44:00.164611 15044 ts_tablet_manager.cc:1939] T a356c7fedd2944a9bff4afb8295f0d89 P 8befea83862149b4a8998d0b68bda268: Deleting consensus metadata
I20260927 00:44:00.166157 14697 catalog_manager.cc:4841] TS 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): tablet a356c7fedd2944a9bff4afb8295f0d89 (table test-table [id=44807019bf364b3c8b209ebea8783704]) successfully deleted
W20260927 00:44:00.247939 15042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:00.248183 15042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:00.248224 15042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:00.252251 15042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:00.252300 15042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:00.252357 15042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:00.257531 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41737
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:00.258423 15042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:00.259332 15042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:44:00.268589 15042 server_base.cc:1034] running on GCE node
W20260927 00:44:00.268610 15049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.268584 15050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.268566 15052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:00.269236 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:00.269838 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:00.271001 15042 hybrid_clock.cc:648] HybridClock initialized: now 1790469840270989 us; error 26 us; skew 500 ppm
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:00.273680 15042 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:00.274493 15042 webserver.cc:460] Webserver started at http://127.0.98.3:41737/ using document root <none> and password file <none>
I20260927 00:44:00.274986 15042 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:00.275055 15042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:00.278713 15042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 00:44:00.280629 15059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:00.281896 15042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:00.282048 15042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b775c329cbd241e8b953de25da0a9ef9"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "b0618973158a4b96677476dc98a24499"
server_key_iv: "437418258df21d42e2fc2ad803a86c9f"
server_key_version: "encryptionkey@0"
I20260927 00:44:00.282557 15042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:00.297260 15042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:00.300451 15042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:00.300597 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:00.301200 15042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:00.301990 15042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:00.302037 15042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:00.302102 15042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:00.302140 15042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:00.310562 15042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46259
I20260927 00:44:00.310642 15172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46259 every 8 connection(s)
I20260927 00:44:00.311558 15042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:00.312170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15042
I20260927 00:44:00.322024 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.312995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.0.98.62:44203)
0927 00:44:00.313179 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.313185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.313470 (+ 285us) server_negotiation.cc:407] Connection header received
0927 00:44:00.313692 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.313695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.313750 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.313841 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:00.314786 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.315550 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.316579 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.316951 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.317066 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.318642 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:00.318649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:00.318651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:00.318678 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:00.320247 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.320685 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.320688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.320689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.320803 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.321168 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.321172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.321173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.321389 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:00.321633 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.321750 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.321866 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:00.327068 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.313498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57871 (local address 127.0.98.62:44203)
0927 00:44:00.313726 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.313730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.314965 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 00:44:00.316023 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.316026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.316076 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.316174 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:00.317733 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.318565 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.320312 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.320689 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.320772 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.323447 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:00.323451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:00.323453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:00.323477 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:00.325085 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.325586 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.325591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.325592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.325683 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:00.326013 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:00.326018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:00.326019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:00.326187 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:00.326294 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.326788 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.326899 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260927 00:44:00.327796 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.314052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:57871)
0927 00:44:00.314804 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.314858 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.315833 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.316371 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.316378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.316850 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:00.317561 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.317582 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.318696 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.318700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.320171 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.320185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.320840 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.320845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.321049 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.321600 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:00.321628 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:00.323300 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:00.325206 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.325211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.325225 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.325481 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.325774 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:00.325779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:00.325781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:00.325916 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:00.326291 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:00.326297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:00.326536 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.326734 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.327031 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":135,"threads_started":1}
I20260927 00:44:00.328825 15173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44203
I20260927 00:44:00.329077 15173 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:00.329643 15173 heartbeater.cc:511] Master 127.0.98.62:44203 requested a full tablet report, sending...
I20260927 00:44:00.330737 14710 ts_manager.cc:194] Registered new tserver with Master: b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259)
I20260927 00:44:00.331267 14710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57871
I20260927 00:44:00.335361 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:00.339594 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.335933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.0.98.62:44203)
0927 00:44:00.336075 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.336079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.336251 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:44:00.336427 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.336430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.336510 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.336613 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:00.336856 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.337404 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.338261 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.338485 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.338593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.338903 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:00.339221 (+ 318us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:00.339351 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.339412 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.339461 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:44:00.340231 14710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:00.343432 14710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:00.350945 14971 tablet_service.cc:1467] Processing CreateTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 (DEFAULT_TABLE table=test-table [id=05807c6b99b545b7a906699553ab2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:00.351609 14971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5e6a0150da42508ff8c0e6ad0907b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:00.354319 14929 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Bootstrap starting.
I20260927 00:44:00.354801 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.350139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46259 (local address 127.0.0.1:49286)
0927 00:44:00.350277 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.350292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.350356 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.351342 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.351346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.351364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:00.351445 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.351451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.352559 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.352563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.353613 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.353623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.354401 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.354405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.354501 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.354535 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.354613 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.354682 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 00:44:00.354859 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.350521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.0.98.3:46259)
0927 00:44:00.350887 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.350896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.350915 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:44:00.350974 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.350977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.351171 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.351324 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:00.351558 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.352378 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.353744 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.354281 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.354405 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.354559 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.354644 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.354736 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":182,"threads_started":1}
I20260927 00:44:00.354883 14832 tablet_service.cc:1467] Processing CreateTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 (DEFAULT_TABLE table=test-table [id=05807c6b99b545b7a906699553ab2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:00.355340 14929 tablet_bootstrap.cc:654] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:00.355459 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5e6a0150da42508ff8c0e6ad0907b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:00.357372 14929 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: No bootstrap required, opened a new log
I20260927 00:44:00.357460 14929 ts_tablet_manager.cc:1397] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:00.357896 14929 raft_consensus.cc:348] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.358039 14929 raft_consensus.cc:374] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:00.358083 14929 raft_consensus.cc:729] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8befea83862149b4a8998d0b68bda268, State: Initialized, Role: FOLLOWER
I20260927 00:44:00.358210 14929 consensus_queue.cc:260] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.358455 14929 ts_tablet_manager.cc:1428] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:00.358457 14790 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap starting.
I20260927 00:44:00.359262 14790 tablet_bootstrap.cc:654] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:00.359449 15107 tablet_service.cc:1467] Processing CreateTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 (DEFAULT_TABLE table=test-table [id=05807c6b99b545b7a906699553ab2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:00.360477 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da5e6a0150da42508ff8c0e6ad0907b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:00.360918 14790 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: No bootstrap required, opened a new log
I20260927 00:44:00.361004 14790 ts_tablet_manager.cc:1397] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 00:44:00.361425 14790 raft_consensus.cc:348] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.361546 14790 raft_consensus.cc:374] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:00.361589 14790 raft_consensus.cc:729] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Initialized, Role: FOLLOWER
I20260927 00:44:00.361707 14790 consensus_queue.cc:260] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.361980 14790 ts_tablet_manager.cc:1428] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:00.367288 15185 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Bootstrap starting.
I20260927 00:44:00.369230 15185 tablet_bootstrap.cc:654] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:00.370294 15185 log.cc:824] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:00.372069 15185 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: No bootstrap required, opened a new log
I20260927 00:44:00.372268 15185 ts_tablet_manager.cc:1397] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:44:00.375033 15185 raft_consensus.cc:348] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.375249 15185 raft_consensus.cc:374] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:00.375283 15185 raft_consensus.cc:729] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b775c329cbd241e8b953de25da0a9ef9, State: Initialized, Role: FOLLOWER
I20260927 00:44:00.375702 15185 consensus_queue.cc:260] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.376355 15173 heartbeater.cc:503] Master 127.0.98.62:44203 was elected leader, sending a full tablet report...
I20260927 00:44:00.376643 15185 ts_tablet_manager.cc:1428] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260927 00:44:00.377272 15039 tablet.cc:2369] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:00.391101 14899 tablet.cc:2369] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:00.562701 15174 tablet.cc:2369] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:00.593631 14904 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:00.593842 14904 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.594962 14904 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:44:00.600376 15043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.595432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46379 (local address 127.0.98.2:33007)
0927 00:44:00.595569 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.595573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.595712 (+ 139us) server_negotiation.cc:407] Connection header received
0927 00:44:00.595825 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.595827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.595890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.595987 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:00.596306 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.597151 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599033 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.599687 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599783 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.599942 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.600200 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.600267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:00.600375 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.595287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36411 (local address 127.0.98.3:46259)
0927 00:44:00.595458 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.595465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.595538 (+ 73us) server_negotiation.cc:407] Connection header received
0927 00:44:00.595762 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.595781 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.595841 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.595933 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:00.596305 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.597145 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599033 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.599636 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599781 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.599845 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.600166 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.600246 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 00:44:00.600469 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.595141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46259 (local address 127.0.98.1:36411)
0927 00:44:00.595455 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.595469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.595590 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.595975 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.595981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.596003 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:00.596183 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.596189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.597269 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.597272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.598889 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.598899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.599804 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.599965 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.600048 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.600128 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.600261 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":246,"spinlock_wait_cycles":1280,"thread_start_us":97,"threads_started":1}
I20260927 00:44:00.600481 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.595275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.98.1:46379)
0927 00:44:00.595628 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.595642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:00.595725 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:00.596011 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:00.596013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:00.596044 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:00.596182 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.596189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.597269 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.597272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.598887 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:00.598899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.599804 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.599807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:00.599991 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.600053 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 00:44:00.600128 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 00:44:00.600265 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":328,"spinlock_wait_cycles":25984,"thread_start_us":64,"threads_started":1}
I20260927 00:44:00.601192 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5e6a0150da42508ff8c0e6ad0907b5" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775c329cbd241e8b953de25da0a9ef9" is_pre_election: true
I20260927 00:44:00.601207 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5e6a0150da42508ff8c0e6ad0907b5" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8befea83862149b4a8998d0b68bda268" is_pre_election: true
I20260927 00:44:00.601504 15127 raft_consensus.cc:2454] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 0.
I20260927 00:44:00.601506 14991 raft_consensus.cc:2454] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 0.
I20260927 00:44:00.601989 14786 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3, b775c329cbd241e8b953de25da0a9ef9; no voters:
I20260927 00:44:00.602437 14904 raft_consensus.cc:2781] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:00.602541 14904 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:00.602591 14904 raft_consensus.cc:3037] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:00.603597 14904 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.603962 14904 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 election: Requested vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:44:00.604283 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5e6a0150da42508ff8c0e6ad0907b5" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775c329cbd241e8b953de25da0a9ef9"
I20260927 00:44:00.604391 14991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da5e6a0150da42508ff8c0e6ad0907b5" candidate_uuid: "422e52334e3b47cf803c90ac72d78ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8befea83862149b4a8998d0b68bda268"
I20260927 00:44:00.604426 15127 raft_consensus.cc:3037] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:00.604552 14991 raft_consensus.cc:3037] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:00.605666 15127 raft_consensus.cc:2454] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 1.
I20260927 00:44:00.605795 14991 raft_consensus.cc:2454] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 422e52334e3b47cf803c90ac72d78ad3 in term 1.
I20260927 00:44:00.605973 14786 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3, b775c329cbd241e8b953de25da0a9ef9; no voters:
I20260927 00:44:00.606153 14904 raft_consensus.cc:2781] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:00.606343 14904 raft_consensus.cc:686] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Running, Role: LEADER
I20260927 00:44:00.606594 14904 consensus_queue.cc:237] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:00.609582 14710 catalog_manager.cc:5495] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "422e52334e3b47cf803c90ac72d78ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:00.650904 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.647129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.1:46633)
0927 00:44:00.647496 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:00.647503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:00.647514 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:00.647584 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:00.647586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:00.647637 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:00.647759 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:00.648029 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.648529 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.649434 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:00.649667 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:00.649780 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:00.650051 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:00.650477 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:00.650610 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:00.650673 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:44:00.650730 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260927 00:44:00.654608 14991 raft_consensus.cc:1261] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Refusing update from remote peer 422e52334e3b47cf803c90ac72d78ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:00.654628 15127 raft_consensus.cc:1261] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Refusing update from remote peer 422e52334e3b47cf803c90ac72d78ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:00.655328 15190 consensus_queue.cc:1035] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:00.655517 14904 consensus_queue.cc:1035] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:00.662288 15194 mvcc.cc:205] Tried to move back new op lower bound from 7333764467315470336 to 7333764467126784000. Current Snapshot: MvccSnapshot[applied={T|T < 7333764467315470336}]
I20260927 00:44:00.664279 15195 mvcc.cc:205] Tried to move back new op lower bound from 7333764467315470336 to 7333764467126784000. Current Snapshot: MvccSnapshot[applied={T|T < 7333764467315470336}]
I20260927 00:44:00.668680 14710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:00.668864 14710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:00.671170 14710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Sending DeleteTablet for 3 replicas of tablet da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:00.672024 14971 tablet_service.cc:1514] Processing DeleteTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45482
I20260927 00:44:00.672073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14748
I20260927 00:44:00.672216 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59832
I20260927 00:44:00.672243 15107 tablet_service.cc:1514] Processing DeleteTablet for tablet da5e6a0150da42508ff8c0e6ad0907b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49286
I20260927 00:44:00.672922 15202 tablet_replica.cc:330] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: stopping tablet replica
I20260927 00:44:00.673100 15202 raft_consensus.cc:2229] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:00.673237 15202 raft_consensus.cc:2258] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:00.673609 15203 tablet_replica.cc:330] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: stopping tablet replica
I20260927 00:44:00.673899 15203 raft_consensus.cc:2229] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:00.674190 15203 raft_consensus.cc:2258] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:00.675053 15202 ts_tablet_manager.cc:1905] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:00.675765 15203 ts_tablet_manager.cc:1905] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:00.677266 15202 ts_tablet_manager.cc:1918] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:00.677371 15202 log.cc:1196] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:00.677614 15202 ts_tablet_manager.cc:1939] T da5e6a0150da42508ff8c0e6ad0907b5 P 8befea83862149b4a8998d0b68bda268: Deleting consensus metadata
I20260927 00:44:00.678459 14697 catalog_manager.cc:4841] TS 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): tablet da5e6a0150da42508ff8c0e6ad0907b5 (table test-table [id=05807c6b99b545b7a906699553ab2324]) successfully deleted
I20260927 00:44:00.679035 15203 ts_tablet_manager.cc:1918] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:00.679152 15203 log.cc:1196] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:00.679502 15203 ts_tablet_manager.cc:1939] T da5e6a0150da42508ff8c0e6ad0907b5 P b775c329cbd241e8b953de25da0a9ef9: Deleting consensus metadata
W20260927 00:44:00.679711 14696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46633 (error 108)
I20260927 00:44:00.680112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14902
I20260927 00:44:00.680812 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:00.680490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46633 (local address 127.0.0.1:59848)
0927 00:44:00.680632 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:00.680715 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 00:44:00.681012 14696 catalog_manager.cc:4584] TS 422e52334e3b47cf803c90ac72d78ad3 (127.0.98.1:46633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da5e6a0150da42508ff8c0e6ad0907b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46633: connect: Connection refused (error 111)
I20260927 00:44:00.681927 14696 catalog_manager.cc:4841] TS b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): tablet da5e6a0150da42508ff8c0e6ad0907b5 (table test-table [id=05807c6b99b545b7a906699553ab2324]) successfully deleted
I20260927 00:44:00.685913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15042
I20260927 00:44:00.692487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 14677
I20260927 00:44:00.699291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--webserver_interface=127.0.98.62
--webserver_port=43883
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:00.807883 15205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:00.808197 15205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:00.808285 15205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:00.812259 15205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:00.812351 15205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:00.812391 15205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:00.812429 15205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:00.812462 15205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:00.817852 15205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:00.818943 15205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:00.819989 15205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:00.828732 15213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.828677 15211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:00.828677 15210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:00.828943 15205 server_base.cc:1034] running on GCE node
I20260927 00:44:00.829694 15205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:00.830808 15205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:00.832015 15205 hybrid_clock.cc:648] HybridClock initialized: now 1790469840832005 us; error 34 us; skew 500 ppm
Sep 27 00:44:00 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:00.834782 15205 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:00.835624 15205 webserver.cc:460] Webserver started at http://127.0.98.62:43883/ using document root <none> and password file <none>
I20260927 00:44:00.836285 15205 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:00.836380 15205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:00.840260 15205 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:00.842315 15220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:00.843556 15205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:00.843664 15205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "c55b813a6b934b01b76ac15752ce10d1"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "50b3a5a1a19008a4e67094ba2ed10ee9"
server_key_iv: "79f090f77ff8cb0d4dea6422745ea9f7"
server_key_version: "encryptionkey@0"
I20260927 00:44:00.844177 15205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:00.854065 15205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:00.856633 15205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:00.856777 15205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:00.863579 15205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44203
I20260927 00:44:00.863862 15272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44203 every 8 connection(s)
I20260927 00:44:00.864646 15205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:00.865054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15205
I20260927 00:44:00.865531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46633
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=36733
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:00.869817 15273 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:44:00.871410 15273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap starting.
I20260927 00:44:00.888185 15273 log.cc:824] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:00.899299 15273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:00.899765 15273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1: Bootstrap complete.
I20260927 00:44:00.903764 15273 raft_consensus.cc:348] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:44:00.904142 15273 raft_consensus.cc:729] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Initialized, Role: FOLLOWER
I20260927 00:44:00.904668 15273 consensus_queue.cc:260] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:44:00.904788 15273 raft_consensus.cc:388] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:00.904876 15273 raft_consensus.cc:482] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:00.905021 15273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:44:00.906867 15273 raft_consensus.cc:504] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:44:00.907253 15273 leader_election.cc:302] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c55b813a6b934b01b76ac15752ce10d1; no voters:
I20260927 00:44:00.907624 15273 leader_election.cc:288] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:44:00.907709 15277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:44:00.907917 15277 raft_consensus.cc:686] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [term 3 LEADER]: Becoming Leader. State: Replica: c55b813a6b934b01b76ac15752ce10d1, State: Running, Role: LEADER
I20260927 00:44:00.908203 15277 consensus_queue.cc:237] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } }
I20260927 00:44:00.908845 15273 sys_catalog.cc:564] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:00.909176 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c55b813a6b934b01b76ac15752ce10d1. Latest consensus state: current_term: 3 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:44:00.909298 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:00.909710 15281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:00.909675 15278 sys_catalog.cc:455] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c55b813a6b934b01b76ac15752ce10d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c55b813a6b934b01b76ac15752ce10d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44203 } } }
I20260927 00:44:00.909958 15278 sys_catalog.cc:458] T 00000000000000000000000000000000 P c55b813a6b934b01b76ac15752ce10d1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:00.914925 15281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:00.915561 15281 catalog_manager.cc:1259] Loaded cluster ID: 0257c21afe3d4c478c12070e6c29099a
I20260927 00:44:00.915624 15281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 00:44:00.916607 15281 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7eed64e9ad google::LogMessage::Fail() at ??:0
@ 0x7f7eed65188a google::LogMessage::SendToLog() at ??:0
@ 0x7f7eed64e4de google::LogMessage::Flush() at ??:0
@ 0x7f7eed64fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ef7f8dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7ef7f8ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7ef7f8b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7ef7fcf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7eef8a03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7eef8a0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7eef8a23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7eef891512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ef0ed56db start_thread at ??:0
@ 0x7f7eec14c61f clone at ??:0
W20260927 00:44:01.002880 15275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:01.003225 15275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:01.003293 15275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:01.009537 15275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:01.009630 15275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:01.009727 15275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:01.017649 15275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36733
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:01.018997 15275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:01.020290 15275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:01.031126 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:01.031200 15275 server_base.cc:1034] running on GCE node
W20260927 00:44:01.031167 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:01.031119 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:01.031821 15275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:01.032439 15275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:01.033619 15275 hybrid_clock.cc:648] HybridClock initialized: now 1790469841033595 us; error 43 us; skew 500 ppm
Sep 27 00:44:01 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:01.036496 15275 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:01.037465 15275 webserver.cc:460] Webserver started at http://127.0.98.1:36733/ using document root <none> and password file <none>
I20260927 00:44:01.038058 15275 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:01.038151 15275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:01.042227 15275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 00:44:01.044380 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.045710 15275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:01.045848 15275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "422e52334e3b47cf803c90ac72d78ad3"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "eec9efeb288976b0cc750640c368e473"
server_key_iv: "12e631102be2bea06639a746baf1daa0"
server_key_version: "encryptionkey@0"
I20260927 00:44:01.046366 15275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:01.122347 15275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:01.126096 15275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:01.126305 15275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:01.127005 15275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:01.128371 15331 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:44:01.132418 15275 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:44:01.132496 15275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 00:44:01.132560 15275 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:44:01.134752 15275 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:44:01.134809 15275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 00:44:01.135125 15331 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap starting.
I20260927 00:44:01.145784 15275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46633
I20260927 00:44:01.145802 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46633 every 8 connection(s)
I20260927 00:44:01.146761 15275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:01.153283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15275
I20260927 00:44:01.153715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33007
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=39871
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:01.156698 15331 log.cc:824] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:01.165552 15331 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:01.165964 15331 tablet_bootstrap.cc:492] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Bootstrap complete.
I20260927 00:44:01.167352 15331 ts_tablet_manager.cc:1397] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.004s
I20260927 00:44:01.171013 15331 raft_consensus.cc:348] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:01.171437 15331 raft_consensus.cc:729] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422e52334e3b47cf803c90ac72d78ad3, State: Initialized, Role: FOLLOWER
I20260927 00:44:01.171936 15331 consensus_queue.cc:260] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:01.172926 15331 ts_tablet_manager.cc:1428] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260927 00:44:01.177881 15441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:44:01.148800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:35317)
0927 00:44:01.149352 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.149404 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:01.150298 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:01.177110 (+ 26812us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":157,"threads_started":1}
W20260927 00:44:01.178442 15439 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44203: Client connection negotiation failed: client connection to 127.0.98.62:44203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:44:01.179103 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.178854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:38869)
0927 00:44:01.178980 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.179050 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 00:44:01.179764 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.179556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:36613)
0927 00:44:01.179677 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.179717 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:44:01.180017 15439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:44:01.264546 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:01.264768 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:01.264807 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:01.268992 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:01.269052 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:01.269163 15442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:01.274312 15442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39871
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:01.275290 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:01.276289 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:01.285713 15449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:01.285804 15442 server_base.cc:1034] running on GCE node
W20260927 00:44:01.285754 15450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:01.285768 15452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:01.286350 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:01.286886 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:01.288066 15442 hybrid_clock.cc:648] HybridClock initialized: now 1790469841288048 us; error 29 us; skew 500 ppm
Sep 27 00:44:01 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:01.290704 15442 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:01.291577 15442 webserver.cc:460] Webserver started at http://127.0.98.2:39871/ using document root <none> and password file <none>
I20260927 00:44:01.292116 15442 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:01.292176 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:01.295894 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:01.297892 15459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.299150 15442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:01.299268 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "8befea83862149b4a8998d0b68bda268"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "5447bf2b822168dd1bd4235004fa14cc"
server_key_iv: "638c483cac8dd86b0ea8d4b43b0b07a9"
server_key_version: "encryptionkey@0"
I20260927 00:44:01.299777 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:01.319279 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:01.322156 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:01.322358 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:01.322952 15442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:01.323743 15442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:01.323791 15442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.323861 15442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:01.323901 15442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.332851 15442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33007
I20260927 00:44:01.332875 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33007 every 8 connection(s)
I20260927 00:44:01.333796 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:01.337157 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.335738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:45761)
0927 00:44:01.336361 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.336462 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":102,"threads_started":1}
W20260927 00:44:01.337592 15573 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44203: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
I20260927 00:44:01.338117 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.337919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:43817)
0927 00:44:01.338014 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.338066 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 00:44:01.338743 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.338566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:46019)
0927 00:44:01.338660 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.338699 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 00:44:01.338974 15573 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:44:01.340986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15442
I20260927 00:44:01.341423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46259
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44203
--webserver_port=41737
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:01.451596 15576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:01.451848 15576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:01.451921 15576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:01.456502 15576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:01.456564 15576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:01.456670 15576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:01.461861 15576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41737
--tserver_master_addrs=127.0.98.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:01.462857 15576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:01.463840 15576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:01.473204 15582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:01.473219 15581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:01.473201 15584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:01.473453 15576 server_base.cc:1034] running on GCE node
I20260927 00:44:01.473925 15576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:01.474529 15576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:01.475687 15576 hybrid_clock.cc:648] HybridClock initialized: now 1790469841475668 us; error 31 us; skew 500 ppm
Sep 27 00:44:01 dist-test-slave-pkws krb5kdc[14130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:01.478325 15576 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:01.479166 15576 webserver.cc:460] Webserver started at http://127.0.98.3:41737/ using document root <none> and password file <none>
I20260927 00:44:01.479705 15576 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:01.479766 15576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:01.483402 15576 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:01.485354 15591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.486507 15576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:01.486639 15576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b775c329cbd241e8b953de25da0a9ef9"
format_stamp: "Formatted at 2026-09-27 00:43:58 on dist-test-slave-pkws"
server_key: "b0618973158a4b96677476dc98a24499"
server_key_iv: "437418258df21d42e2fc2ad803a86c9f"
server_key_version: "encryptionkey@0"
I20260927 00:44:01.487169 15576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:01.502668 15576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:01.505434 15576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:01.505645 15576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:01.506177 15576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:01.506947 15576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:01.507017 15576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.507082 15576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:01.507130 15576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:01.515800 15576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46259
I20260927 00:44:01.515823 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46259 every 8 connection(s)
I20260927 00:44:01.516719 15576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:01.517612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15576
I20260927 00:44:01.520561 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.519203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:60535)
0927 00:44:01.519806 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.519903 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":105,"threads_started":1}
W20260927 00:44:01.521096 15705 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44203: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
I20260927 00:44:01.521665 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.521452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:53053)
0927 00:44:01.521556 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.521603 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 00:44:01.522270 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:01.522094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:58827)
0927 00:44:01.522185 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:01.522224 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:44:01.522491 15705 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:44:02.181031 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.180580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:33625)
0927 00:44:02.180813 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:02.180865 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260927 00:44:02.339830 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.339464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:40673)
0927 00:44:02.339694 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:02.339742 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":62,"threads_started":1}
I20260927 00:44:02.470768 15711 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:02.471010 15711 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:02.472047 15711 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
Sep 27 00:44:02 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:44:02 dist-test-slave-pkws krb5kdc[14130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:44:02.486402 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.472315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33007 (local address 127.0.98.1:57165)
0927 00:44:02.472526 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:02.472544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:02.472667 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:02.474318 (+ 1651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:02.474328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:02.474909 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:02.475429 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:02.475445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.476739 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.476746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:02.477797 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:02.477807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.478511 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.478518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:02.478719 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:02.479435 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:02.479448 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:02.481565 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:02.483907 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:02.483909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:02.483911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:02.484154 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:02.484557 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:02.484565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:02.484660 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:02.485523 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:02.485525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:02.485673 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 00:44:02.485871 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 00:44:02.486200 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":122,"spinlock_wait_cycles":64512}
I20260927 00:44:02.486402 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.472315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46259 (local address 127.0.98.1:36899)
0927 00:44:02.472914 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:02.472930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:02.473034 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:02.474430 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:02.474433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:02.475167 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:02.475428 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:02.475444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.476833 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.476835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:02.477793 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:02.477802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.478569 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.478572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:02.478715 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:02.479362 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:02.479377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:02.481156 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:02.483575 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:02.483581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:02.483607 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:02.483975 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:02.484361 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:02.484367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:02.484499 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:02.485363 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:02.485369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:02.485668 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 00:44:02.485871 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:44:02.486252 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":286,"spinlock_wait_cycles":2048,"thread_start_us":72,"threads_started":1}
I20260927 00:44:02.486505 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.472461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57165 (local address 127.0.98.2:33007)
0927 00:44:02.472831 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:02.472852 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:02.472931 (+ 79us) server_negotiation.cc:407] Connection header received
0927 00:44:02.473599 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:02.473604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:02.473948 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:02.474185 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:02.475613 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.476575 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.477950 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.478385 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.478537 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:02.481763 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:02.481768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:02.481780 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:02.481807 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:02.483792 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484279 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:02.484285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:02.484291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:02.484457 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484757 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:02.484760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:02.484762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:02.485234 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:02.485384 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:02.486102 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 00:44:02.486332 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":72,"threads_started":1}
I20260927 00:44:02.487329 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.472490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36899 (local address 127.0.98.3:46259)
0927 00:44:02.472943 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:02.472964 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:02.473011 (+ 47us) server_negotiation.cc:407] Connection header received
0927 00:44:02.473789 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:02.473794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:02.474145 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:02.474468 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:02.475603 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.476701 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.477920 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:02.478419 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:02.478612 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:02.481394 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:02.481399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:02.481410 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:02.481435 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:02.483450 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484110 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:02.484116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:02.484121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:02.484262 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:02.484647 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:02.484650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:02.484651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:02.485234 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:02.485384 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:02.486853 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0927 00:44:02.487133 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":120,"threads_started":1}
W20260927 00:44:02.489368 15327 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
W20260927 00:44:02.489526 15328 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:02.489614 15328 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3; no voters: 8befea83862149b4a8998d0b68bda268, b775c329cbd241e8b953de25da0a9ef9
I20260927 00:44:02.489835 15711 raft_consensus.cc:2726] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:44:02.523351 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:02.522976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:37767)
0927 00:44:02.523214 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:02.523264 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260927 00:44:03.182289 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:03.181842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:54825)
0927 00:44:03.182136 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:03.182186 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260927 00:44:03.340979 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:03.340570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:45867)
0927 00:44:03.340798 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:03.340846 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 00:44:03.524678 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:03.524185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:59565)
0927 00:44:03.524487 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:03.524581 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260927 00:44:04.118336 15719 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:04.118466 15719 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:04.118911 15719 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
W20260927 00:44:04.119697 15328 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
W20260927 00:44:04.119818 15327 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:04.119925 15327 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3; no voters: 8befea83862149b4a8998d0b68bda268, b775c329cbd241e8b953de25da0a9ef9
I20260927 00:44:04.120090 15719 raft_consensus.cc:2726] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:44:04.183447 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:04.183074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:43361)
0927 00:44:04.183301 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:04.183354 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260927 00:44:04.342008 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:04.341620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:46065)
0927 00:44:04.341861 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:04.341914 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260927 00:44:04.525678 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:04.525285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:55599)
0927 00:44:04.525542 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:04.525594 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260927 00:44:05.184496 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:05.184126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:46205)
0927 00:44:05.184350 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:05.184402 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260927 00:44:05.343132 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:05.342699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:36249)
0927 00:44:05.342987 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:05.343052 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260927 00:44:05.526774 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:05.526376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:49921)
0927 00:44:05.526628 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:05.526681 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260927 00:44:06.184537 15726 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:06.184665 15726 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:06.185148 15726 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
I20260927 00:44:06.185851 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:06.185406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:32799)
0927 00:44:06.185658 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:06.185737 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":65,"threads_started":1}
W20260927 00:44:06.185926 15328 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
W20260927 00:44:06.186015 15327 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:06.186095 15327 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3; no voters: 8befea83862149b4a8998d0b68bda268, b775c329cbd241e8b953de25da0a9ef9
I20260927 00:44:06.186244 15726 raft_consensus.cc:2726] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:44:06.344378 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:06.343992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:40117)
0927 00:44:06.344243 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:06.344295 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260927 00:44:06.527870 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:06.527465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:50737)
0927 00:44:06.527724 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:06.527774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260927 00:44:07.186978 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:07.186572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:33021)
0927 00:44:07.186831 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:07.186885 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260927 00:44:07.345520 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:07.345134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:43433)
0927 00:44:07.345380 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:07.345432 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260927 00:44:07.528883 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:07.528494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:46247)
0927 00:44:07.528747 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:07.528800 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260927 00:44:08.188077 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:08.187687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:48703)
0927 00:44:08.187933 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:08.187985 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260927 00:44:08.346643 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:08.346238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:49349)
0927 00:44:08.346498 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:08.346550 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
I20260927 00:44:08.530084 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:08.529651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:44203)
0927 00:44:08.529915 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:08.529997 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":68,"threads_started":1}
I20260927 00:44:09.189178 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:09.188777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:38043)
0927 00:44:09.189028 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:09.189088 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260927 00:44:09.347664 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:09.347295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:34479)
0927 00:44:09.347523 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:09.347573 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260927 00:44:09.531093 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:09.530731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:48635)
0927 00:44:09.530959 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:09.531008 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260927 00:44:10.190331 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:10.189844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:59391)
0927 00:44:10.190147 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:10.190201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260927 00:44:10.348992 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:10.348492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:50847)
0927 00:44:10.348746 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:10.348800 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260927 00:44:10.503645 15741 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:10.503798 15741 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:10.504251 15741 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
W20260927 00:44:10.504993 15328 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
W20260927 00:44:10.505147 15327 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:10.505223 15327 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3; no voters: 8befea83862149b4a8998d0b68bda268, b775c329cbd241e8b953de25da0a9ef9
I20260927 00:44:10.505378 15741 raft_consensus.cc:2726] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:44:10.532115 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:10.531744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:33339)
0927 00:44:10.531972 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:10.532034 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
I20260927 00:44:11.191582 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:11.191157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:38727)
0927 00:44:11.191421 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:11.191490 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260927 00:44:11.350231 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:11.349716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:55795)
0927 00:44:11.350009 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:11.350103 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":140,"threads_started":1}
I20260927 00:44:11.533252 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:11.532840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:47129)
0927 00:44:11.533117 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:11.533165 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20260927 00:44:12.192608 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:12.192229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:38503)
0927 00:44:12.192457 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:12.192514 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260927 00:44:12.351423 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:12.350986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:35659)
0927 00:44:12.351275 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:12.351328 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
I20260927 00:44:12.534404 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:12.534018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:50631)
0927 00:44:12.534263 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:12.534313 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260927 00:44:12.902598 15749 raft_consensus.cc:482] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:12.902716 15749 raft_consensus.cc:504] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b775c329cbd241e8b953de25da0a9ef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46259 } } peers { permanent_uuid: "422e52334e3b47cf803c90ac72d78ad3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46633 } } peers { permanent_uuid: "8befea83862149b4a8998d0b68bda268" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33007 } }
I20260927 00:44:12.903214 15749 leader_election.cc:288] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259), 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007)
W20260927 00:44:12.903971 15327 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b775c329cbd241e8b953de25da0a9ef9 (127.0.98.3:46259): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
W20260927 00:44:12.904044 15328 leader_election.cc:341] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8befea83862149b4a8998d0b68bda268 (127.0.98.2:33007): Not found: Tablet not found: da5e6a0150da42508ff8c0e6ad0907b5
I20260927 00:44:12.904085 15327 leader_election.cc:302] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 422e52334e3b47cf803c90ac72d78ad3; no voters: 8befea83862149b4a8998d0b68bda268, b775c329cbd241e8b953de25da0a9ef9
I20260927 00:44:12.904229 15749 raft_consensus.cc:2726] T da5e6a0150da42508ff8c0e6ad0907b5 P 422e52334e3b47cf803c90ac72d78ad3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:44:13.193784 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:13.193360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:41149)
0927 00:44:13.193619 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:13.193676 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":111,"threads_started":1}
I20260927 00:44:13.352483 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:13.352083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:59467)
0927 00:44:13.352328 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:13.352395 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260927 00:44:13.535614 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:13.535225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:48523)
0927 00:44:13.535462 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:13.535526 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260927 00:44:14.194873 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:14.194471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:33879)
0927 00:44:14.194726 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:14.194780 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260927 00:44:14.353577 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:14.353176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:38225)
0927 00:44:14.353433 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:14.353487 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260927 00:44:14.536755 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:14.536362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:55565)
0927 00:44:14.536598 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:14.536648 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
I20260927 00:44:15.195942 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:15.195558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:50185)
0927 00:44:15.195794 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:15.195846 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260927 00:44:15.354774 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:15.354334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:33375)
0927 00:44:15.354616 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:15.354664 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":110,"threads_started":1}
I20260927 00:44:15.537797 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:15.537422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.3:33535)
0927 00:44:15.537656 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:15.537707 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260927 00:44:16.197577 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.197087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.1:47021)
0927 00:44:16.197377 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:16.197449 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260927 00:44:16.355962 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:16.355564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44203 (local address 127.0.98.2:49663)
0927 00:44:16.355815 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:16.355871 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
I20260927 00:44:16.522957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15275
I20260927 00:44:16.528474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15442
I20260927 00:44:16.533582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15576
2026-09-27T00:44:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19573 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:44:16 dist-test-slave-pkws krb5kdc[15763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:44:16 dist-test-slave-pkws krb5kdc[15763](info): set up 2 sockets
Sep 27 00:44:16 dist-test-slave-pkws krb5kdc[15763](info): commencing operation
krb5kdc: starting...
W20260927 00:44:17.569626 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:44:17 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:44:17.693275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:17.799474 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:17.799715 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:17.799754 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:17.803591 15776 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:44:17.803632 15776 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:44:17.803647 15776 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:44:17.803660 15776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:17.803678 15776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:17.803696 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:17.803711 15776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:17.803726 15776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:17.808617 15776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:17.809547 15776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:17.810422 15776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:44:17.819024 15776 server_base.cc:1034] running on GCE node
W20260927 00:44:17.819008 15781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:17.819062 15784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:17.818986 15782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:17.819856 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:17.820950 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:17.822146 15776 hybrid_clock.cc:648] HybridClock initialized: now 1790469857822132 us; error 37 us; skew 500 ppm
Sep 27 00:44:17 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:17.824869 15776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:17.826110 15776 webserver.cc:460] Webserver started at http://127.0.98.62:44209/ using document root <none> and password file <none>
I20260927 00:44:17.826869 15776 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:17.826980 15776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:17.827309 15776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:17.829942 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "d230faed973c4e59b9a4a11a6d043918"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-pkws"
server_key: "ea4946ba36dfef83dc23513b6f8faca4"
server_key_iv: "95f41ff0cc5333e5e71d31c1a0093bf8"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.830580 15776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "d230faed973c4e59b9a4a11a6d043918"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-pkws"
server_key: "ea4946ba36dfef83dc23513b6f8faca4"
server_key_iv: "95f41ff0cc5333e5e71d31c1a0093bf8"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.834978 15776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 00:44:17.837414 15791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:17.838529 15776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:17.838667 15776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "d230faed973c4e59b9a4a11a6d043918"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-pkws"
server_key: "ea4946ba36dfef83dc23513b6f8faca4"
server_key_iv: "95f41ff0cc5333e5e71d31c1a0093bf8"
server_key_version: "encryptionkey@0"
I20260927 00:44:17.838819 15776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:17.855172 15776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:17.858264 15776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:17.858460 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:17.864918 15776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36079
I20260927 00:44:17.865231 15843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36079 every 8 connection(s)
I20260927 00:44:17.865833 15776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:17.868368 15844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:17.869258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15776
I20260927 00:44:17.869410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:44:17.869688 392 external_mini_cluster.cc:1428] Setting key c0636c901cf5c5a9f6097b1145a5868e
I20260927 00:44:17.874758 15844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap starting.
I20260927 00:44:17.877175 15844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:17.878108 15844 log.cc:824] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Log is configured to *not* fsync() on all Append() calls
Sep 27 00:44:17 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:17.880187 15844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: No bootstrap required, opened a new log
I20260927 00:44:17.884608 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:17.871034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.98.62:36079)
0927 00:44:17.871438 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:17.871457 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:17.871500 (+ 43us) server_negotiation.cc:407] Connection header received
0927 00:44:17.872303 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:17.872326 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:17.872715 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:17.873103 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:17.873763 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:17.875179 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:17.875997 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:17.876646 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:17.876819 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:17.879632 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:17.879638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:17.879653 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:17.879682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:17.881558 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:17.881957 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:17.881962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:17.881968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:17.882097 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:17.882348 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:17.882352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:17.882353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:17.882975 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:17.883165 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:17.883372 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 00:44:17.883668 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260927 00:44:17.884816 15844 raft_consensus.cc:348] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:17.885083 15844 raft_consensus.cc:374] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:17.885140 15844 raft_consensus.cc:729] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Initialized, Role: FOLLOWER
I20260927 00:44:17.885586 15844 consensus_queue.cc:260] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:17.885763 15844 raft_consensus.cc:388] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:17.885829 15844 raft_consensus.cc:482] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:17.885921 15844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:17.886866 15844 raft_consensus.cc:504] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:17.887197 15844 leader_election.cc:302] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d230faed973c4e59b9a4a11a6d043918; no voters:
I20260927 00:44:17.887490 15844 leader_election.cc:288] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:44:17.887593 15849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:17.887789 15849 raft_consensus.cc:686] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 LEADER]: Becoming Leader. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Running, Role: LEADER
I20260927 00:44:17.888247 15849 consensus_queue.cc:237] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:17.888713 15844 sys_catalog.cc:564] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:17.889847 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d230faed973c4e59b9a4a11a6d043918. Latest consensus state: current_term: 1 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:44:17.889976 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:17.890312 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:44:17.890439 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:17.890390 15858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:17.893895 15858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:17.899019 15858 catalog_manager.cc:1347] Generated new cluster ID: c5492c60978846d1b7fcb9cd489fc6da
I20260927 00:44:17.899080 15858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:17.915967 15858 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:44:17.916976 15858 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:17.923003 15858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Generated new TSK 0
I20260927 00:44:17.923749 15858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:44:17.958015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.065896 15870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.066186 15870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.066260 15870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.071192 15870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.071259 15870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.071350 15870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:18.076959 15870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:18.077997 15870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.079015 15870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.088527 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.088531 15876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.088542 15875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.088773 15870 server_base.cc:1034] running on GCE node
I20260927 00:44:18.089218 15870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.089895 15870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.091073 15870 hybrid_clock.cc:648] HybridClock initialized: now 1790469858091058 us; error 26 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.093799 15870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:18.094647 15870 webserver.cc:460] Webserver started at http://127.0.98.1:33675/ using document root <none> and password file <none>
I20260927 00:44:18.095196 15870 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.095283 15870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.095512 15870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.097549 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1d8f696be144e42af056bf67e4c69af"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "96f1082eb58387c23c230b9e9ef22286"
server_key_iv: "4264178b57b8b517c535362b7882b66c"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.098127 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1d8f696be144e42af056bf67e4c69af"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "96f1082eb58387c23c230b9e9ef22286"
server_key_iv: "4264178b57b8b517c535362b7882b66c"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.101765 15870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:18.104158 15885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.105387 15870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:18.105518 15870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "d1d8f696be144e42af056bf67e4c69af"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "96f1082eb58387c23c230b9e9ef22286"
server_key_iv: "4264178b57b8b517c535362b7882b66c"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.105628 15870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.117427 15870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.120575 15870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.120762 15870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.121387 15870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.122169 15870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.122239 15870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.122308 15870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.122355 15870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.130947 15870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41325
I20260927 00:44:18.130995 15998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41325 every 8 connection(s)
I20260927 00:44:18.131832 15870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:18.134023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 15870
I20260927 00:44:18.134152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:44:18.134447 392 external_mini_cluster.cc:1428] Setting key bcdb22049fa9ade8160921b4b4d808ac
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:44:18.145752 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.133694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38537 (local address 127.0.98.62:36079)
0927 00:44:18.133949 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.133955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.134738 (+ 783us) server_negotiation.cc:407] Connection header received
0927 00:44:18.135996 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.135999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.136048 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.136133 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.137403 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.137960 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.139340 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.139576 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.139690 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.142183 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.142189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.142196 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.142219 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.143684 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.144185 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.144190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.144192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.144267 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.144590 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.144594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.144596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.144836 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.144977 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.145451 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.145582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:44:18.146608 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.134052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:38537)
0927 00:44:18.134576 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.134626 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.135806 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.136259 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.136265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.136668 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.137201 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.137214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.138088 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.138093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.139173 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.139182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.139722 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.139725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.139901 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.140489 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.140505 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.142033 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.143807 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.143812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.143824 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.144085 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.144364 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.144368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.144370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.144498 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.144975 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.144982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.145207 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.145379 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.145691 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":158,"threads_started":1}
I20260927 00:44:18.147853 15999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:18.148075 15999 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.148528 15999 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:18.149978 15808 ts_manager.cc:194] Registered new tserver with Master: d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325)
I20260927 00:44:18.151079 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:44:18.161170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.267112 16004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.267344 16004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.267386 16004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.271229 16004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.271273 16004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.271328 16004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:18.276204 16004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:18.277139 16004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.278049 16004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.287352 16010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.287438 16009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.287400 16012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.287672 16004 server_base.cc:1034] running on GCE node
I20260927 00:44:18.288048 16004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.288661 16004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.289814 16004 hybrid_clock.cc:648] HybridClock initialized: now 1790469858289791 us; error 33 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.292459 16004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:18.293341 16004 webserver.cc:460] Webserver started at http://127.0.98.2:42847/ using document root <none> and password file <none>
I20260927 00:44:18.293887 16004 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.293998 16004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.294237 16004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.296268 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffab519300974d25bbd878ac933804c1"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "0d9a4dc8168d4b7549ba8a6c40a99164"
server_key_iv: "9c2bfd14b1d7f6c6974f94fba8c56c4b"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.296864 16004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffab519300974d25bbd878ac933804c1"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "0d9a4dc8168d4b7549ba8a6c40a99164"
server_key_iv: "9c2bfd14b1d7f6c6974f94fba8c56c4b"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.300410 16004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:18.302736 16019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.303891 16004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:18.304024 16004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "ffab519300974d25bbd878ac933804c1"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "0d9a4dc8168d4b7549ba8a6c40a99164"
server_key_iv: "9c2bfd14b1d7f6c6974f94fba8c56c4b"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.304131 16004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.324721 16004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.328022 16004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.328224 16004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.328780 16004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.329587 16004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.329658 16004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.329730 16004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.329773 16004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.338219 16004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36379
I20260927 00:44:18.338243 16132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36379 every 8 connection(s)
I20260927 00:44:18.339044 16004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:18.346663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16004
I20260927 00:44:18.346807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:44:18.347095 392 external_mini_cluster.cc:1428] Setting key 27b067e23ca7615f6390a0466a83bb4e
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:18.352722 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.340707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44151 (local address 127.0.98.62:36079)
0927 00:44:18.340847 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.340850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.341701 (+ 851us) server_negotiation.cc:407] Connection header received
0927 00:44:18.343000 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.343005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.343081 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.343204 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.344292 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.344817 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.345949 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.346154 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.346227 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.348699 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.348703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.348705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.348732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.350395 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.351058 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.351061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.351062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.351129 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.351517 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.351519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.351520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.351768 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.351867 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.352331 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.352466 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:44:18.353308 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.341094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:44151)
0927 00:44:18.341523 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.341563 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.342773 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.343368 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.343374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.343706 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.344172 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.344184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.344963 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.344966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.345847 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.345855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.346267 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.346269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.346397 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.346968 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.346983 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.348545 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.350530 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.350539 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.350554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.350941 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.351214 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.351220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.351225 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.351419 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.351866 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.351871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.352093 (+ 222us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.352242 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.352473 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260927 00:44:18.354405 16133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:18.354650 16133 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.355099 16133 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:18.356108 15808 ts_manager.cc:194] Registered new tserver with Master: ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
I20260927 00:44:18.356665 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44151
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:44:18.372761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:18.478919 16138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:18.479168 16138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:18.479238 16138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:18.483170 16138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:18.483248 16138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:18.483363 16138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:18.488509 16138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:18.489547 16138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:18.490513 16138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:18.499571 16143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.499651 16138 server_base.cc:1034] running on GCE node
W20260927 00:44:18.499572 16144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:18.499554 16146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:18.500133 16138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:18.500771 16138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:18.501963 16138 hybrid_clock.cc:648] HybridClock initialized: now 1790469858501932 us; error 45 us; skew 500 ppm
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:18.504467 16138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:18.505371 16138 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260927 00:44:18.505921 16138 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:18.506009 16138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:18.506229 16138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:44:18.508404 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "202d3e80699a436bbfb831ffa108a053"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "973b645d01a7b6fc66011af4a7e4dc93"
server_key_iv: "b66d92b0ade231e61f6494e5bcefdf05"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.509038 16138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "202d3e80699a436bbfb831ffa108a053"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "973b645d01a7b6fc66011af4a7e4dc93"
server_key_iv: "b66d92b0ade231e61f6494e5bcefdf05"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.512508 16138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:44:18.514829 16153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.515964 16138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:18.516093 16138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "202d3e80699a436bbfb831ffa108a053"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "973b645d01a7b6fc66011af4a7e4dc93"
server_key_iv: "b66d92b0ade231e61f6494e5bcefdf05"
server_key_version: "encryptionkey@0"
I20260927 00:44:18.516197 16138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:18.525444 16138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:18.528018 16138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:18.528200 16138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:18.528710 16138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:18.529518 16138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:18.529587 16138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.529656 16138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:18.529699 16138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:18.538239 16138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42979
I20260927 00:44:18.538264 16266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42979 every 8 connection(s)
I20260927 00:44:18.539041 16138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:18.548081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16138
Sep 27 00:44:18 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:18.548203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:44:18.548475 392 external_mini_cluster.cc:1428] Setting key bd114e772b8d9cd64c2b30de8dcef6b9
I20260927 00:44:18.552070 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.540713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55093 (local address 127.0.98.62:36079)
0927 00:44:18.540852 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.540856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.541721 (+ 865us) server_negotiation.cc:407] Connection header received
0927 00:44:18.543040 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.543044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.543101 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.543207 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.544311 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.544836 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.546035 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.546259 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.546369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.548899 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.548905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.548910 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.548952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.550393 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.550860 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.550863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.550864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.550945 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.551222 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.551224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.551226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.551384 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.551460 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.551860 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.551956 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:44:18.552842 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.541068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:55093)
0927 00:44:18.541488 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.541527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.542827 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.543372 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.543378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.543718 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:18.544155 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.544167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.545036 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.545041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.545901 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.545910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.546414 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.546418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.546591 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.547078 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:18.547093 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:18.548702 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:18.550499 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.550507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.550519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.550771 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.551025 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:18.551027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:18.551028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:18.551146 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:18.551466 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:18.551470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:18.551668 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.551814 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.552034 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260927 00:44:18.553869 16267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:18.554085 16267 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:18.554522 16267 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:18.555447 15808 ts_manager.cc:194] Registered new tserver with Master: 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979)
I20260927 00:44:18.555965 15808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55093
I20260927 00:44:18.562031 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:18.571105 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.563945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.0.98.62:36079)
0927 00:44:18.564080 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.564086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.564283 (+ 197us) server_negotiation.cc:407] Connection header received
0927 00:44:18.564437 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.564440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.564479 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.564560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:18.565283 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.565778 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.566596 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.566808 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.566916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.568196 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:18.568205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:18.568207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:18.568228 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:18.569459 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.569879 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.569884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.569886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.569970 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:18.570255 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:18.570258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:18.570260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:18.570466 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:18.570606 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.570829 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.570949 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:18.573679 15808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:18.576558 15808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:18.590875 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.584695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36379 (local address 127.0.0.1:40234)
0927 00:44:18.585202 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.585246 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.585367 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.586138 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.586145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.586183 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.586309 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.586316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.587731 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.587735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.589237 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.589246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590290 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.590411 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.590455 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.590572 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.590736 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":162,"thread_start_us":57,"threads_started":1}
I20260927 00:44:18.590904 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.584884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.0.98.2:36379)
0927 00:44:18.585310 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.585323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.585349 (+ 26us) server_negotiation.cc:407] Connection header received
0927 00:44:18.585567 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.585577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.585794 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.585943 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.586436 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.587584 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.589354 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590164 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590271 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.590415 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.590617 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.590755 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":87,"threads_started":1}
I20260927 00:44:18.591271 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.585486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.0.1:59312)
0927 00:44:18.585867 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.585879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.585959 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.586796 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.586798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.586810 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.586882 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.586891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.588052 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.588055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.589580 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.589591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590840 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.590945 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.590978 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.591052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.591118 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":136,"thread_start_us":88,"threads_started":1}
I20260927 00:44:18.591584 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.584893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41325 (local address 127.0.0.1:33342)
0927 00:44:18.585495 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.585509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.585605 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.586715 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.586721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.586742 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.586855 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.586863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.588093 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.588096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.589580 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.589626 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590967 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.591070 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.591100 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.591169 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.591241 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":197,"thread_start_us":99,"threads_started":1}
I20260927 00:44:18.591583 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.585459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.0.98.1:41325)
0927 00:44:18.586229 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.586241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.586263 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:44:18.586343 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.586347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.586546 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.586700 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.586971 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.587944 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.589754 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590629 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590770 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.590903 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.591345 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.591453 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":148,"threads_started":1}
I20260927 00:44:18.592597 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.585883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.98.3:42979)
0927 00:44:18.586375 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.586390 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.586414 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:44:18.586485 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.586489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.586624 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.587020 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.587130 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.587912 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.589725 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.590523 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.590644 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.590771 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.592410 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.592496 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
I20260927 00:44:18.594245 16067 tablet_service.cc:1467] Processing CreateTablet for tablet 354b1755cede4c5185371e8008a75e0f (DEFAULT_TABLE table=test-table [id=ab13bf8f2eac4893907d155b31b10090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.594245 15933 tablet_service.cc:1467] Processing CreateTablet for tablet 354b1755cede4c5185371e8008a75e0f (DEFAULT_TABLE table=test-table [id=ab13bf8f2eac4893907d155b31b10090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.594774 16201 tablet_service.cc:1467] Processing CreateTablet for tablet 354b1755cede4c5185371e8008a75e0f (DEFAULT_TABLE table=test-table [id=ab13bf8f2eac4893907d155b31b10090]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:18.595372 15933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b1755cede4c5185371e8008a75e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.595372 16067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b1755cede4c5185371e8008a75e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.595768 16201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b1755cede4c5185371e8008a75e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:18.601468 16284 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Bootstrap starting.
I20260927 00:44:18.602819 16285 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Bootstrap starting.
I20260927 00:44:18.603669 16284 tablet_bootstrap.cc:654] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.604507 16285 tablet_bootstrap.cc:654] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.604724 16284 log.cc:824] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.605244 16286 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Bootstrap starting.
I20260927 00:44:18.605748 16285 log.cc:824] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.607074 16284 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: No bootstrap required, opened a new log
I20260927 00:44:18.607255 16284 ts_tablet_manager.cc:1397] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:44:18.607615 16285 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: No bootstrap required, opened a new log
I20260927 00:44:18.607620 16286 tablet_bootstrap.cc:654] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:18.607788 16285 ts_tablet_manager.cc:1397] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 00:44:18.608455 16286 log.cc:824] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:18.610203 16286 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: No bootstrap required, opened a new log
I20260927 00:44:18.610394 16286 ts_tablet_manager.cc:1397] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:44:18.611055 16284 raft_consensus.cc:348] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.611322 16284 raft_consensus.cc:374] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.611385 16284 raft_consensus.cc:729] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202d3e80699a436bbfb831ffa108a053, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.611464 16285 raft_consensus.cc:348] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.611680 16285 raft_consensus.cc:374] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.611739 16285 raft_consensus.cc:729] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8f696be144e42af056bf67e4c69af, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.611829 16284 consensus_queue.cc:260] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.612128 16285 consensus_queue.cc:260] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.612517 16267 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
I20260927 00:44:18.613202 16284 ts_tablet_manager.cc:1428] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:44:18.613399 16285 ts_tablet_manager.cc:1428] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 00:44:18.613747 15999 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
I20260927 00:44:18.613930 16286 raft_consensus.cc:348] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.614174 16286 raft_consensus.cc:374] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:18.614208 16286 raft_consensus.cc:729] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffab519300974d25bbd878ac933804c1, State: Initialized, Role: FOLLOWER
I20260927 00:44:18.614584 16286 consensus_queue.cc:260] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.615185 16133 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
I20260927 00:44:18.615458 16286 ts_tablet_manager.cc:1428] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 00:44:18.632977 16000 tablet.cc:2369] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:18.707859 16292 raft_consensus.cc:482] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:18.708060 16292 raft_consensus.cc:504] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.709363 16292 leader_election.cc:288] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325), 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979)
I20260927 00:44:18.713105 16135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.709545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41325 (local address 127.0.98.2:40307)
0927 00:44:18.709718 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.709733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.709830 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.710123 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.710126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.710143 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.710267 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.710276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.711082 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.711085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.711952 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.711961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.712635 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.712746 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.712784 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.712857 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.712964 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 00:44:18.713109 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.709595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40307 (local address 127.0.98.1:41325)
0927 00:44:18.709738 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.709744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.709884 (+ 140us) server_negotiation.cc:407] Connection header received
0927 00:44:18.709947 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.709950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.710003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.710095 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.710386 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.710970 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712074 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.712511 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712589 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.712664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.712906 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.712985 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:44:18.713135 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.709749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56199 (local address 127.0.98.3:42979)
0927 00:44:18.709889 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.709893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.710094 (+ 201us) server_negotiation.cc:407] Connection header received
0927 00:44:18.710187 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.710191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.710244 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.710324 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:18.710595 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.711119 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712144 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.712584 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712688 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.712748 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.712960 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.713030 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 00:44:18.713137 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.709593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.98.2:56199)
0927 00:44:18.710000 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.710013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:18.710106 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:18.710359 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:18.710362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:18.710380 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:18.710491 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.710497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.711224 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.711226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.711997 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:18.712030 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.712705 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.712709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:18.712801 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.712818 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 00:44:18.712944 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 00:44:18.713006 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":121,"threads_started":1}
I20260927 00:44:18.713790 16221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b1755cede4c5185371e8008a75e0f" candidate_uuid: "ffab519300974d25bbd878ac933804c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202d3e80699a436bbfb831ffa108a053" is_pre_election: true
I20260927 00:44:18.713981 15953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b1755cede4c5185371e8008a75e0f" candidate_uuid: "ffab519300974d25bbd878ac933804c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8f696be144e42af056bf67e4c69af" is_pre_election: true
I20260927 00:44:18.714131 16221 raft_consensus.cc:2454] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffab519300974d25bbd878ac933804c1 in term 0.
I20260927 00:44:18.714282 15953 raft_consensus.cc:2454] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffab519300974d25bbd878ac933804c1 in term 0.
I20260927 00:44:18.714656 16022 leader_election.cc:302] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1; no voters:
I20260927 00:44:18.714933 16292 raft_consensus.cc:2781] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:18.715014 16292 raft_consensus.cc:482] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:18.715056 16292 raft_consensus.cc:3037] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.716012 16292 raft_consensus.cc:504] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.716316 16292 leader_election.cc:288] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [CANDIDATE]: Term 1 election: Requested vote from peers d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325), 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979)
I20260927 00:44:18.716601 15953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b1755cede4c5185371e8008a75e0f" candidate_uuid: "ffab519300974d25bbd878ac933804c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8f696be144e42af056bf67e4c69af"
I20260927 00:44:18.716603 16221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b1755cede4c5185371e8008a75e0f" candidate_uuid: "ffab519300974d25bbd878ac933804c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202d3e80699a436bbfb831ffa108a053"
I20260927 00:44:18.716737 15953 raft_consensus.cc:3037] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.716754 16221 raft_consensus.cc:3037] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:18.717705 15953 raft_consensus.cc:2454] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffab519300974d25bbd878ac933804c1 in term 1.
I20260927 00:44:18.717747 16221 raft_consensus.cc:2454] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffab519300974d25bbd878ac933804c1 in term 1.
I20260927 00:44:18.718022 16022 leader_election.cc:302] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1; no voters:
I20260927 00:44:18.718179 16292 raft_consensus.cc:2781] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:18.718364 16292 raft_consensus.cc:686] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 1 LEADER]: Becoming Leader. State: Replica: ffab519300974d25bbd878ac933804c1, State: Running, Role: LEADER
I20260927 00:44:18.718695 16292 consensus_queue.cc:237] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:18.721860 15807 catalog_manager.cc:5495] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffab519300974d25bbd878ac933804c1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ffab519300974d25bbd878ac933804c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } health_report { overall_health: UNKNOWN } } }
I20260927 00:44:18.738953 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.735609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.0.98.2:36379)
0927 00:44:18.735724 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:18.735727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:18.735739 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:44:18.735840 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:18.735842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:18.735889 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:18.735964 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:18.736177 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.736644 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.737561 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:18.737774 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:18.737888 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:18.738185 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:18.738557 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:18.738683 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:18.738743 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 00:44:18.738801 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 00:44:18.744251 15953 raft_consensus.cc:1261] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Refusing update from remote peer ffab519300974d25bbd878ac933804c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:18.744251 16221 raft_consensus.cc:1261] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Refusing update from remote peer ffab519300974d25bbd878ac933804c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:18.744940 16294 consensus_queue.cc:1035] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:18.745143 16292 consensus_queue.cc:1035] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:18.761677 15807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:18.761831 15807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:18.764132 15807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Sending DeleteTablet for 3 replicas of tablet 354b1755cede4c5185371e8008a75e0f
I20260927 00:44:18.764853 16067 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b1755cede4c5185371e8008a75e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40234
I20260927 00:44:18.764910 15933 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b1755cede4c5185371e8008a75e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33342
I20260927 00:44:18.765027 16201 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b1755cede4c5185371e8008a75e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59312
I20260927 00:44:18.765386 16310 tablet_replica.cc:330] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: stopping tablet replica
I20260927 00:44:18.765501 16308 tablet_replica.cc:330] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: stopping tablet replica
I20260927 00:44:18.765532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15870
I20260927 00:44:18.766366 16308 raft_consensus.cc:2229] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:18.766834 16308 raft_consensus.cc:2258] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:18.767745 16310 raft_consensus.cc:2229] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:18.768038 16310 raft_consensus.cc:2258] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:18.768128 16308 ts_tablet_manager.cc:1905] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:18.769300 16310 ts_tablet_manager.cc:1905] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:18.770895 16308 ts_tablet_manager.cc:1918] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:18.770988 16308 log.cc:1196] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/354b1755cede4c5185371e8008a75e0f
I20260927 00:44:18.771304 16308 ts_tablet_manager.cc:1939] T 354b1755cede4c5185371e8008a75e0f P ffab519300974d25bbd878ac933804c1: Deleting consensus metadata
I20260927 00:44:18.772378 15792 catalog_manager.cc:4841] TS ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379): tablet 354b1755cede4c5185371e8008a75e0f (table test-table [id=ab13bf8f2eac4893907d155b31b10090]) successfully deleted
I20260927 00:44:18.772537 16310 ts_tablet_manager.cc:1918] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:18.772622 16310 log.cc:1196] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/354b1755cede4c5185371e8008a75e0f
I20260927 00:44:18.772982 16310 ts_tablet_manager.cc:1939] T 354b1755cede4c5185371e8008a75e0f P 202d3e80699a436bbfb831ffa108a053: Deleting consensus metadata
I20260927 00:44:18.774075 15794 catalog_manager.cc:4841] TS 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): tablet 354b1755cede4c5185371e8008a75e0f (table test-table [id=ab13bf8f2eac4893907d155b31b10090]) successfully deleted
W20260927 00:44:18.774541 15793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41325 (error 108)
I20260927 00:44:18.775398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16004
I20260927 00:44:18.775574 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:18.775279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41325 (local address 127.0.0.1:33358)
0927 00:44:18.775393 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:18.775475 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 00:44:18.776432 15793 catalog_manager.cc:4584] TS d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 354b1755cede4c5185371e8008a75e0f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41325: connect: Connection refused (error 111)
I20260927 00:44:18.780272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16138
I20260927 00:44:18.784994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 15776
I20260927 00:44:38.790201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--webserver_interface=127.0.98.62
--webserver_port=44209
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:44:38.896466 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:38.896704 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:38.896768 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:38.900811 16312 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:44:38.900857 16312 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:44:38.900873 16312 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:44:38.900923 16312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:44:38.900964 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:38.901001 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:38.901037 16312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:44:38.901082 16312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:44:38.906141 16312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:38.907068 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:38.908037 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:38.920454 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:38.920428 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:38.920488 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:38.920645 16312 server_base.cc:1034] running on GCE node
I20260927 00:44:38.921327 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:38.922420 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:38.923662 16312 hybrid_clock.cc:648] HybridClock initialized: now 1790469878923631 us; error 34 us; skew 500 ppm
Sep 27 00:44:38 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:38.932293 16312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:44:38.933267 16312 webserver.cc:460] Webserver started at http://127.0.98.62:44209/ using document root <none> and password file <none>
I20260927 00:44:38.933846 16312 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:38.933936 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:38.937729 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:44:38.939879 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:38.941067 16312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:38.941187 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "d230faed973c4e59b9a4a11a6d043918"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-pkws"
server_key: "ea4946ba36dfef83dc23513b6f8faca4"
server_key_iv: "95f41ff0cc5333e5e71d31c1a0093bf8"
server_key_version: "encryptionkey@0"
I20260927 00:44:38.941751 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:38.954862 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:38.957849 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:38.958037 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:38.964937 16312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36079
I20260927 00:44:38.965234 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36079 every 8 connection(s)
I20260927 00:44:38.965956 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:44:38.966173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16312
I20260927 00:44:38.966646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=33675
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:44:38.971132 16381 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:44:38.973654 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap starting.
I20260927 00:44:38.996074 16381 log.cc:824] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.006524 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:39.006937 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap complete.
I20260927 00:44:39.012387 16381 raft_consensus.cc:348] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:39.012925 16381 raft_consensus.cc:729] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.013454 16381 consensus_queue.cc:260] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:39.013605 16381 raft_consensus.cc:388] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:44:39.013677 16381 raft_consensus.cc:482] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:44:39.013772 16381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 1 FOLLOWER]: Advancing to term 2
I20260927 00:44:39.016175 16381 raft_consensus.cc:504] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:39.016534 16381 leader_election.cc:302] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d230faed973c4e59b9a4a11a6d043918; no voters:
I20260927 00:44:39.016846 16381 leader_election.cc:288] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 00:44:39.017064 16385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 00:44:39.017388 16385 raft_consensus.cc:686] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 LEADER]: Becoming Leader. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Running, Role: LEADER
I20260927 00:44:39.017735 16385 consensus_queue.cc:237] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:44:39.018091 16381 sys_catalog.cc:564] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:44:39.019048 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d230faed973c4e59b9a4a11a6d043918. Latest consensus state: current_term: 2 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:44:39.019173 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:39.019503 16386 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:44:39.019584 16386 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:44:39.020121 16394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:44:39.025045 16394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:44:39.025636 16394 catalog_manager.cc:1259] Loaded cluster ID: c5492c60978846d1b7fcb9cd489fc6da
I20260927 00:44:39.025693 16394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:44:39.028258 16394 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:44:39.028944 16394 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Loaded TSK: 0
I20260927 00:44:39.048720 16394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Generated new TSK 1
I20260927 00:44:39.049005 16394 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:44:39.098909 16383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.099139 16383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.099179 16383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.103327 16383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.103379 16383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.103471 16383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:44:39.108503 16383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33675
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:39.109493 16383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.110431 16383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.119748 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.119760 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.120023 16383 server_base.cc:1034] running on GCE node
W20260927 00:44:39.119794 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.120545 16383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.121213 16383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.122390 16383 hybrid_clock.cc:648] HybridClock initialized: now 1790469879122363 us; error 38 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.125003 16383 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:44:39.125875 16383 webserver.cc:460] Webserver started at http://127.0.98.1:33675/ using document root <none> and password file <none>
I20260927 00:44:39.126400 16383 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.126477 16383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.130192 16383 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 00:44:39.132258 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.133513 16383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:39.133621 16383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "d1d8f696be144e42af056bf67e4c69af"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "96f1082eb58387c23c230b9e9ef22286"
server_key_iv: "4264178b57b8b517c535362b7882b66c"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.134130 16383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.151953 16383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.155113 16383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.155303 16383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.155906 16383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.157037 16425 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:44:39.160796 16383 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:44:39.160852 16383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260927 00:44:39.160935 16383 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:44:39.162563 16383 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:44:39.162618 16383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:44:39.162859 16425 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Bootstrap starting.
I20260927 00:44:39.173316 16383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41325
I20260927 00:44:39.173398 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41325 every 8 connection(s)
I20260927 00:44:39.174471 16383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:44:39.182665 16425 log.cc:824] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.183862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16383
I20260927 00:44:39.184392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36379
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=42847
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.190716 16425 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:44:39.191123 16425 tablet_bootstrap.cc:492] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Bootstrap complete.
I20260927 00:44:39.191633 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.176477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44301 (local address 127.0.98.62:36079)
0927 00:44:39.176836 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.176856 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.177202 (+ 346us) server_negotiation.cc:407] Connection header received
0927 00:44:39.178284 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.178293 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.178577 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.178898 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.179999 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.181032 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.182180 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.182641 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.182816 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.185704 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.185712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.185726 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.185757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.187846 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.188556 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.188562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.188567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.188723 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.189163 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.189167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.189168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.189582 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.189710 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.190434 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.190746 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
I20260927 00:44:39.191690 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.176508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:44301)
0927 00:44:39.177039 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.177081 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.177873 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.178955 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.178962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.179378 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.179825 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.179838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.181204 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.181207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.182052 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.182061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.182785 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.182789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.182984 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.183622 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.183650 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.185493 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.187979 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.187988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.188004 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.188422 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.188832 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.188837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.188842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.189055 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.189708 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.189717 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.190052 (+ 335us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.190283 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.190649 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260927 00:44:39.192452 16425 ts_tablet_manager.cc:1397] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.005s
I20260927 00:44:39.194222 16533 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:39.194437 16533 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.194921 16533 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:39.194953 16425 raft_consensus.cc:348] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:39.195468 16425 raft_consensus.cc:729] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8f696be144e42af056bf67e4c69af, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.195919 16425 consensus_queue.cc:260] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } } peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260927 00:44:39.196919 16425 ts_tablet_manager.cc:1428] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
I20260927 00:44:39.198771 16345 ts_manager.cc:194] Registered new tserver with Master: d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325)
I20260927 00:44:39.202507 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44301
I20260927 00:44:39.204756 16533 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 00:44:39.212580 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.201834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41325 (local address 127.0.0.1:40760)
0927 00:44:39.202317 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.202358 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.202477 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.203096 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.203101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.203485 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.203630 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.203638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.205124 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.205130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.205842 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.205851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.206516 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.206531 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.206644 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.207152 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.207163 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.208472 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.211114 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.211121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.211125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.211333 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.211624 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.211626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.211627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.211697 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.212141 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.212145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.212233 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.212336 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.212466 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":111,"threads_started":1}
I20260927 00:44:39.212733 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.201880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.1:41325)
0927 00:44:39.202734 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.202742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.202759 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:44:39.202806 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.202809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.202939 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.203092 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.203751 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.205002 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.205970 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.206418 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.206515 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.208606 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.208615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.208626 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.208657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.211015 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.211435 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.211440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.211442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.211535 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.211774 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.211777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.211779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.212034 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.212168 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.212431 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.212584 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":768,"thread_start_us":101,"threads_started":1}
I20260927 00:44:39.213590 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b1755cede4c5185371e8008a75e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260927 00:44:39.214332 16542 tablet_replica.cc:330] stopping tablet replica
I20260927 00:44:39.214609 16542 raft_consensus.cc:2229] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:39.214823 16542 raft_consensus.cc:2258] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:39.215649 16542 ts_tablet_manager.cc:1905] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:39.219156 16542 ts_tablet_manager.cc:1918] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:39.219228 16542 log.cc:1196] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal/wals/354b1755cede4c5185371e8008a75e0f
I20260927 00:44:39.219496 16542 ts_tablet_manager.cc:1939] T 354b1755cede4c5185371e8008a75e0f P d1d8f696be144e42af056bf67e4c69af: Deleting consensus metadata
I20260927 00:44:39.221136 16330 catalog_manager.cc:4841] TS d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325): tablet 354b1755cede4c5185371e8008a75e0f (table test-table [id=ab13bf8f2eac4893907d155b31b10090]) successfully deleted
W20260927 00:44:39.313694 16538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.313935 16538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.314002 16538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.318081 16538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.318137 16538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.318243 16538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:44:39.323540 16538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42847
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:39.324517 16538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.325567 16538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.334987 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.335007 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.335093 16538 server_base.cc:1034] running on GCE node
W20260927 00:44:39.334949 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.335501 16538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.336107 16538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.337284 16538 hybrid_clock.cc:648] HybridClock initialized: now 1790469879337267 us; error 29 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.340137 16538 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:44:39.341002 16538 webserver.cc:460] Webserver started at http://127.0.98.2:42847/ using document root <none> and password file <none>
I20260927 00:44:39.341530 16538 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.341646 16538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.345552 16538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:44:39.347561 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.348753 16538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:39.348876 16538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "ffab519300974d25bbd878ac933804c1"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "0d9a4dc8168d4b7549ba8a6c40a99164"
server_key_iv: "9c2bfd14b1d7f6c6974f94fba8c56c4b"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.349421 16538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.363694 16538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.366811 16538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.367000 16538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.367538 16538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.368323 16538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:39.368392 16538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.368456 16538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:39.368507 16538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.377121 16538 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36379
I20260927 00:44:39.377167 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36379 every 8 connection(s)
I20260927 00:44:39.378095 16538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:44:39.380702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16538
I20260927 00:44:39.381187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42979
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=39331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.393558 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.379785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54293 (local address 127.0.98.62:36079)
0927 00:44:39.379938 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.379943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.381283 (+ 1340us) server_negotiation.cc:407] Connection header received
0927 00:44:39.382571 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.382578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.382649 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.382714 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.384049 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.384989 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.386119 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.386574 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.386727 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.389564 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.389571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.389577 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.389615 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.391430 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.391988 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.391994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.391997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.392097 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.392426 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.392430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.392432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.392653 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.392752 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.393253 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.393381 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:39.394316 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.380149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:54293)
0927 00:44:39.381131 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.381184 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.382275 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.383086 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.383093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.383437 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.383891 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.383904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.385131 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.385134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.385994 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.386003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.386738 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.386740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.386878 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.387447 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.387463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.389385 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.391577 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.391585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.391602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.391882 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.392202 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.392205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.392206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.392332 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.392751 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.392756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.393028 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.393194 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.393437 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":631,"threads_started":1}
I20260927 00:44:39.395506 16671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:39.395757 16671 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.396343 16671 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:39.397410 16345 ts_manager.cc:194] Registered new tserver with Master: ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
I20260927 00:44:39.397960 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54293
W20260927 00:44:39.498267 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:44:39.498520 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:44:39.498591 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:44:39.503079 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:44:39.503135 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:44:39.503248 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:44:39.508738 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39331
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:44:39.509733 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:44:39.510746 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:44:39.520429 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.520458 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:44:39.520458 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:44:39.520678 16674 server_base.cc:1034] running on GCE node
I20260927 00:44:39.521123 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:44:39.521731 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:44:39.522907 16674 hybrid_clock.cc:648] HybridClock initialized: now 1790469879522890 us; error 26 us; skew 500 ppm
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:44:39.525568 16674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:44:39.526463 16674 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260927 00:44:39.527025 16674 fs_manager.cc:359] Metadata directory not provided
I20260927 00:44:39.527129 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:44:39.530735 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:44:39.532591 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.533766 16674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:44:39.533900 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "202d3e80699a436bbfb831ffa108a053"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "973b645d01a7b6fc66011af4a7e4dc93"
server_key_iv: "b66d92b0ade231e61f6494e5bcefdf05"
server_key_version: "encryptionkey@0"
I20260927 00:44:39.534443 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:44:39.549577 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:44:39.552248 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:44:39.552412 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:44:39.552982 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:44:39.553818 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:44:39.553884 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.553944 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:44:39.553990 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:44:39.562417 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42979
I20260927 00:44:39.562443 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42979 every 8 connection(s)
I20260927 00:44:39.563442 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:44:39.567252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16674
Sep 27 00:44:39 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:44:39.576530 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.567865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.0.98.62:36079)
0927 00:44:39.568094 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.568098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.568245 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:44:39.568347 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.568350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.568391 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.568475 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.569214 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.569856 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.571035 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.571436 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.571542 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.573178 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.573183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.573188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.573217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.574792 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.575251 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.575257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.575260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.575359 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.575655 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.575657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.575658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.575817 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.575991 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.576207 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.576323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":161,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260927 00:44:39.577971 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.565074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36095 (local address 127.0.98.62:36079)
0927 00:44:39.565206 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.565210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.565992 (+ 782us) server_negotiation.cc:407] Connection header received
0927 00:44:39.567288 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.567292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.567348 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.567451 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.568855 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.569580 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.571118 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.571465 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.571549 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.574378 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.574381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.574384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.574412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.575910 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.576494 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.576497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.576498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.576565 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.576996 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.576999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.577001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.577151 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.577254 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.577719 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.577818 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":51,"spinlock_wait_cycles":1792}
I20260927 00:44:39.578771 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.565440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:36095)
0927 00:44:39.565852 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.565892 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.567075 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.567625 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.567631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.568081 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:44:39.568721 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.568736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.569701 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.569704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.571012 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.571025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.571595 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.571599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.571785 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.572280 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:44:39.572303 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:44:39.574214 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:44:39.576025 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.576030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.576043 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.576386 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.576700 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:44:39.576703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:44:39.576705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:44:39.576863 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:44:39.577254 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:44:39.577260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:44:39.577493 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.577664 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.577962 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260927 00:44:39.579715 16803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36079
I20260927 00:44:39.579929 16803 heartbeater.cc:464] Registering TS with master...
I20260927 00:44:39.580381 16803 heartbeater.cc:511] Master 127.0.98.62:36079 requested a full tablet report, sending...
I20260927 00:44:39.581321 16345 ts_manager.cc:194] Registered new tserver with Master: 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979)
I20260927 00:44:39.581859 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36095
I20260927 00:44:39.589783 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:44:39.593755 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.590358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.0.98.62:36079)
0927 00:44:39.590479 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.590483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.590689 (+ 206us) server_negotiation.cc:407] Connection header received
0927 00:44:39.590862 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.590864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.590944 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.591060 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:39.591279 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.591752 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.592627 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.592845 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.593008 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.593282 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:39.593393 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 00:44:39.593634 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60616: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 00:44:39.593833 16535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60616: token expired
W20260927 00:44:39.594100 16270 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36079)
I20260927 00:44:39.600795 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.594434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.0.98.62:36079)
0927 00:44:39.594531 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.594536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.594585 (+ 49us) server_negotiation.cc:407] Connection header received
0927 00:44:39.594690 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.594694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.594733 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.594819 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:44:39.595413 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.595844 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.596636 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.596832 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.597019 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.598170 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:44:39.598174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:44:39.598176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:44:39.598199 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:44:39.599306 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.599708 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.599712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.599714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.599789 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:44:39.600040 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:44:39.600044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:44:39.600046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:44:39.600199 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:44:39.600328 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.600549 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.600653 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:44:39.603042 16345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:44:39.606179 16345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:44:39.613047 16467 tablet_service.cc:1467] Processing CreateTablet for tablet c10a05e82cae4edcbf570a50b27a1778 (DEFAULT_TABLE table=test-table [id=0bb43721902f451ca6b0343705a4a2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.613711 16467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a05e82cae4edcbf570a50b27a1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.616499 16425 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Bootstrap starting.
I20260927 00:44:39.617398 16425 tablet_bootstrap.cc:654] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.617425 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.611958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.0.1:42632)
0927 00:44:39.612228 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.612245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.612324 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.613482 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.613485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.613502 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.613604 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.613610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.614897 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.614901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.615999 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.616009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.617074 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.617077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.617182 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.617205 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.617251 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.617319 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 00:44:39.617659 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.612615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.98.2:36379)
0927 00:44:39.613081 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.613091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.613108 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:44:39.613168 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.613171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.613366 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.613521 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.613922 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.614628 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.615762 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.616806 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.616986 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.617147 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.617210 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.617301 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":136,"threads_started":1}
I20260927 00:44:39.617429 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.612138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36379 (local address 127.0.0.1:54896)
0927 00:44:39.612672 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.612686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.612780 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.613683 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.613685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.613703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.613796 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.613804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.614770 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.614775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.615644 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.615653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.616968 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.616972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.617063 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.617109 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.617172 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.617241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":98,"threads_started":1}
I20260927 00:44:39.618157 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.612112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.0.98.3:42979)
0927 00:44:39.612963 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.612976 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.613000 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:44:39.613099 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.613104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.613312 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.613470 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.613721 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.614764 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.616148 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.616948 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.617638 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.617841 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.617892 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.617985 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":166,"threads_started":1}
I20260927 00:44:39.618988 16425 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: No bootstrap required, opened a new log
I20260927 00:44:39.619110 16425 ts_tablet_manager.cc:1397] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 00:44:39.619509 16425 raft_consensus.cc:348] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.619616 16425 raft_consensus.cc:374] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.619649 16425 raft_consensus.cc:729] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8f696be144e42af056bf67e4c69af, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.619767 16425 consensus_queue.cc:260] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.620016 16425 ts_tablet_manager.cc:1428] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 00:44:39.620896 16605 tablet_service.cc:1467] Processing CreateTablet for tablet c10a05e82cae4edcbf570a50b27a1778 (DEFAULT_TABLE table=test-table [id=0bb43721902f451ca6b0343705a4a2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.621345 16737 tablet_service.cc:1467] Processing CreateTablet for tablet c10a05e82cae4edcbf570a50b27a1778 (DEFAULT_TABLE table=test-table [id=0bb43721902f451ca6b0343705a4a2ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:44:39.622040 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a05e82cae4edcbf570a50b27a1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.622321 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c10a05e82cae4edcbf570a50b27a1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:44:39.628261 16816 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Bootstrap starting.
I20260927 00:44:39.628705 16817 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Bootstrap starting.
I20260927 00:44:39.630528 16817 tablet_bootstrap.cc:654] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.630874 16816 tablet_bootstrap.cc:654] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Neither blocks nor log segments found. Creating new log.
I20260927 00:44:39.631528 16817 log.cc:824] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.631537 16816 log.cc:824] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:44:39.633165 16816 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: No bootstrap required, opened a new log
I20260927 00:44:39.633174 16817 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: No bootstrap required, opened a new log
I20260927 00:44:39.633339 16816 ts_tablet_manager.cc:1397] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 00:44:39.633350 16817 ts_tablet_manager.cc:1397] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 00:44:39.636200 16817 raft_consensus.cc:348] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.636200 16816 raft_consensus.cc:348] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.636409 16817 raft_consensus.cc:374] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.636433 16816 raft_consensus.cc:374] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:44:39.636471 16817 raft_consensus.cc:729] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202d3e80699a436bbfb831ffa108a053, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.636495 16816 raft_consensus.cc:729] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffab519300974d25bbd878ac933804c1, State: Initialized, Role: FOLLOWER
I20260927 00:44:39.636971 16817 consensus_queue.cc:260] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.636972 16816 consensus_queue.cc:260] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.637805 16671 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
I20260927 00:44:39.637806 16803 heartbeater.cc:503] Master 127.0.98.62:36079 was elected leader, sending a full tablet report...
I20260927 00:44:39.637912 16817 ts_tablet_manager.cc:1428] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260927 00:44:39.637921 16816 ts_tablet_manager.cc:1428] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260927 00:44:39.675686 16534 tablet.cc:2369] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:44:39.814450 16804 tablet.cc:2369] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:39.830884 16539 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:44:39.831156 16539 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.832687 16539 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
I20260927 00:44:39.836917 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.832981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.98.1:40125)
0927 00:44:39.833215 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.833246 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.833375 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.833675 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.833676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.833711 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.833845 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.833852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.834733 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.834737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.835671 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.835680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.836384 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.836388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.836499 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.836557 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.836633 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.836702 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260927 00:44:39.836943 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.833016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40125 (local address 127.0.98.3:42979)
0927 00:44:39.833167 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.833173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.833338 (+ 165us) server_negotiation.cc:407] Connection header received
0927 00:44:39.833517 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.833523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.833581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.833674 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.833959 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.834621 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.835786 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.836275 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.836354 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.836403 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.836659 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.836737 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:44:39.837023 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.833817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46231 (local address 127.0.98.2:36379)
0927 00:44:39.833955 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.833959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.833972 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:44:39.834027 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.834029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.834066 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.834127 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:44:39.834357 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.834954 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.835983 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.836425 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.836528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.836587 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.836790 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.836854 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:44:39.837178 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.833413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36379 (local address 127.0.98.1:46231)
0927 00:44:39.833843 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.833853 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:44:39.833938 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:44:39.834151 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:44:39.834154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:44:39.834172 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:44:39.834257 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.834262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.835052 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.835054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.835859 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:44:39.835872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.836539 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.836542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:44:39.836630 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.836692 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 00:44:39.836764 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 00:44:39.836829 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":67,"threads_started":1}
I20260927 00:44:39.837513 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a05e82cae4edcbf570a50b27a1778" candidate_uuid: "d1d8f696be144e42af056bf67e4c69af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202d3e80699a436bbfb831ffa108a053" is_pre_election: true
I20260927 00:44:39.837698 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a05e82cae4edcbf570a50b27a1778" candidate_uuid: "d1d8f696be144e42af056bf67e4c69af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffab519300974d25bbd878ac933804c1" is_pre_election: true
I20260927 00:44:39.837867 16757 raft_consensus.cc:2454] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d8f696be144e42af056bf67e4c69af in term 0.
I20260927 00:44:39.837939 16625 raft_consensus.cc:2454] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d8f696be144e42af056bf67e4c69af in term 0.
I20260927 00:44:39.838368 16421 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202d3e80699a436bbfb831ffa108a053, d1d8f696be144e42af056bf67e4c69af; no voters:
I20260927 00:44:39.838654 16539 raft_consensus.cc:2781] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:44:39.838786 16539 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:44:39.838845 16539 raft_consensus.cc:3037] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.839902 16539 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.840232 16539 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 1 election: Requested vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
I20260927 00:44:39.840543 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a05e82cae4edcbf570a50b27a1778" candidate_uuid: "d1d8f696be144e42af056bf67e4c69af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffab519300974d25bbd878ac933804c1"
I20260927 00:44:39.840669 16625 raft_consensus.cc:3037] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.841805 16625 raft_consensus.cc:2454] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d8f696be144e42af056bf67e4c69af in term 1.
I20260927 00:44:39.842068 16757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c10a05e82cae4edcbf570a50b27a1778" candidate_uuid: "d1d8f696be144e42af056bf67e4c69af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202d3e80699a436bbfb831ffa108a053"
I20260927 00:44:39.842131 16419 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1d8f696be144e42af056bf67e4c69af, ffab519300974d25bbd878ac933804c1; no voters:
I20260927 00:44:39.842206 16757 raft_consensus.cc:3037] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:44:39.842317 16539 raft_consensus.cc:2781] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:44:39.842505 16539 raft_consensus.cc:686] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 LEADER]: Becoming Leader. State: Replica: d1d8f696be144e42af056bf67e4c69af, State: Running, Role: LEADER
I20260927 00:44:39.842774 16539 consensus_queue.cc:237] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:44:39.843451 16757 raft_consensus.cc:2454] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d8f696be144e42af056bf67e4c69af in term 1.
I20260927 00:44:39.845709 16343 catalog_manager.cc:5495] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af reported cstate change: term changed from 0 to 1, leader changed from <none> to d1d8f696be144e42af056bf67e4c69af (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d1d8f696be144e42af056bf67e4c69af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } health_report { overall_health: HEALTHY } } }
W20260927 00:44:39.879220 16672 tablet.cc:2369] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:44:39.912496 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.908860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.98.1:41325)
0927 00:44:39.909213 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 00:44:39.909220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:44:39.909231 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:44:39.909303 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:44:39.909305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:44:39.909357 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:44:39.909488 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:44:39.909726 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.910230 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.911068 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:44:39.911300 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:44:39.911407 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:44:39.911669 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:44:39.912089 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:44:39.912211 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:44:39.912268 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 00:44:39.912320 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
I20260927 00:44:39.916417 16757 raft_consensus.cc:1261] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Refusing update from remote peer d1d8f696be144e42af056bf67e4c69af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:39.916419 16625 raft_consensus.cc:1261] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Refusing update from remote peer d1d8f696be144e42af056bf67e4c69af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:44:39.917152 16824 consensus_queue.cc:1035] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:39.917343 16539 consensus_queue.cc:1035] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [LEADER]: Connected to new peer: Peer: permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:44:39.924396 16829 mvcc.cc:205] Tried to move back new op lower bound from 7333764628130934784 to 7333764627838980096. Current Snapshot: MvccSnapshot[applied={T|T < 7333764628130934784}]
I20260927 00:44:39.924499 16830 mvcc.cc:205] Tried to move back new op lower bound from 7333764628130934784 to 7333764627838980096. Current Snapshot: MvccSnapshot[applied={T|T < 7333764628130934784}]
I20260927 00:44:39.931519 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:39.931672 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:44:39.933808 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Sending DeleteTablet for 3 replicas of tablet c10a05e82cae4edcbf570a50b27a1778
I20260927 00:44:39.934471 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet c10a05e82cae4edcbf570a50b27a1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42632
I20260927 00:44:39.934474 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet c10a05e82cae4edcbf570a50b27a1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260927 00:44:39.934520 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet c10a05e82cae4edcbf570a50b27a1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:44:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260927 00:44:39.934907 16838 tablet_replica.cc:330] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: stopping tablet replica
I20260927 00:44:39.935045 16838 raft_consensus.cc:2229] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:44:39.935140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16383
I20260927 00:44:39.935930 16839 tablet_replica.cc:330] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: stopping tablet replica
I20260927 00:44:39.936230 16839 raft_consensus.cc:2229] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:39.936524 16839 raft_consensus.cc:2258] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:39.936913 16840 tablet_replica.cc:330] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: stopping tablet replica
I20260927 00:44:39.937208 16840 raft_consensus.cc:2229] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:44:39.937477 16840 raft_consensus.cc:2258] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:44:39.938625 16839 ts_tablet_manager.cc:1905] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:39.938633 16840 ts_tablet_manager.cc:1905] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:44:39.941277 16840 ts_tablet_manager.cc:1918] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:39.941277 16839 ts_tablet_manager.cc:1918] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:44:39.941377 16839 log.cc:1196] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/c10a05e82cae4edcbf570a50b27a1778
I20260927 00:44:39.941377 16840 log.cc:1196] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/c10a05e82cae4edcbf570a50b27a1778
I20260927 00:44:39.941721 16840 ts_tablet_manager.cc:1939] T c10a05e82cae4edcbf570a50b27a1778 P 202d3e80699a436bbfb831ffa108a053: Deleting consensus metadata
I20260927 00:44:39.941721 16839 ts_tablet_manager.cc:1939] T c10a05e82cae4edcbf570a50b27a1778 P ffab519300974d25bbd878ac933804c1: Deleting consensus metadata
W20260927 00:44:39.942245 16330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41325 (error 108)
I20260927 00:44:39.942627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16538
I20260927 00:44:39.943341 16331 catalog_manager.cc:4841] TS 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): tablet c10a05e82cae4edcbf570a50b27a1778 (table test-table [id=0bb43721902f451ca6b0343705a4a2ea]) successfully deleted
I20260927 00:44:39.944303 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.943974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41325 (local address 127.0.0.1:40774)
0927 00:44:39.944124 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.944204 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 00:44:39.944455 16330 catalog_manager.cc:4584] TS d1d8f696be144e42af056bf67e4c69af (127.0.98.1:41325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c10a05e82cae4edcbf570a50b27a1778: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41325: connect: Connection refused (error 111)
I20260927 00:44:39.949280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16674
I20260927 00:44:39.949677 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:44:39.949361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36379 (local address 127.0.0.1:54900)
0927 00:44:39.949532 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 00:44:39.949589 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 00:44:39.954855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16312
I20260927 00:44:59.961272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--webserver_interface=127.0.98.62
--webserver_port=44209
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:00.068333 16842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:00.068579 16842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:00.068619 16842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:00.072595 16842 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 00:45:00.072644 16842 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 00:45:00.072660 16842 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 00:45:00.072674 16842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:00.072724 16842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:00.072760 16842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:00.072781 16842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:00.072806 16842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:00.078004 16842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:00.078991 16842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:00.079967 16842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:00.088358 16847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.088373 16848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.088346 16850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.088675 16842 server_base.cc:1034] running on GCE node
I20260927 00:45:00.089375 16842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:00.090452 16842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:00.091643 16842 hybrid_clock.cc:648] HybridClock initialized: now 1790469900091631 us; error 31 us; skew 500 ppm
Sep 27 00:45:00 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:00.094391 16842 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:00.095237 16842 webserver.cc:460] Webserver started at http://127.0.98.62:44209/ using document root <none> and password file <none>
I20260927 00:45:00.095782 16842 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:00.095870 16842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:00.099521 16842 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:00.101619 16857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.102768 16842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:00.102905 16842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "d230faed973c4e59b9a4a11a6d043918"
format_stamp: "Formatted at 2026-09-27 00:44:17 on dist-test-slave-pkws"
server_key: "ea4946ba36dfef83dc23513b6f8faca4"
server_key_iv: "95f41ff0cc5333e5e71d31c1a0093bf8"
server_key_version: "encryptionkey@0"
I20260927 00:45:00.103430 16842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:00.145660 16842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:00.149490 16842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:00.149736 16842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:00.157562 16842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36079
I20260927 00:45:00.157934 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36079 every 8 connection(s)
I20260927 00:45:00.158658 16842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:00.159344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16842
I20260927 00:45:00.159796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41325
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=33675
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:00.164471 16910 sys_catalog.cc:263] Verifying existing consensus state
I20260927 00:45:00.166837 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap starting.
I20260927 00:45:00.185232 16910 log.cc:824] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:00.196123 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:45:00.196507 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Bootstrap complete.
I20260927 00:45:00.200471 16910 raft_consensus.cc:348] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:45:00.200846 16910 raft_consensus.cc:729] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Initialized, Role: FOLLOWER
I20260927 00:45:00.201347 16910 consensus_queue.cc:260] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:45:00.201462 16910 raft_consensus.cc:388] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:00.201560 16910 raft_consensus.cc:482] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:00.201685 16910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 2 FOLLOWER]: Advancing to term 3
I20260927 00:45:00.203471 16910 raft_consensus.cc:504] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:45:00.203790 16910 leader_election.cc:302] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d230faed973c4e59b9a4a11a6d043918; no voters:
I20260927 00:45:00.204104 16910 leader_election.cc:288] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 00:45:00.204181 16914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 00:45:00.204398 16914 raft_consensus.cc:686] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [term 3 LEADER]: Becoming Leader. State: Replica: d230faed973c4e59b9a4a11a6d043918, State: Running, Role: LEADER
I20260927 00:45:00.204777 16914 consensus_queue.cc:237] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } }
I20260927 00:45:00.205339 16910 sys_catalog.cc:564] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:00.205657 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d230faed973c4e59b9a4a11a6d043918. Latest consensus state: current_term: 3 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:45:00.205780 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:00.206143 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d230faed973c4e59b9a4a11a6d043918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d230faed973c4e59b9a4a11a6d043918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36079 } } }
I20260927 00:45:00.206305 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:00.206389 16918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:00.211282 16918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:00.211947 16918 catalog_manager.cc:1259] Loaded cluster ID: c5492c60978846d1b7fcb9cd489fc6da
I20260927 00:45:00.212020 16918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:00.213903 16918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:00.214478 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Loaded TSK: 0
I20260927 00:45:00.214562 16918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d230faed973c4e59b9a4a11a6d043918: Loaded TSK: 1
F20260927 00:45:00.215139 16918 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3f68dcb9ad google::LogMessage::Fail() at ??:0
@ 0x7f3f68dce88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3f68dcb4de google::LogMessage::Flush() at ??:0
@ 0x7f3f68dcca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3f6ba8f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3f6ba87f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3f737307bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3f7370a99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3f7370b178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3f7374cdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3f7376b82b std::function<>::operator()() at ??:0
@ 0x7f3f7370ab4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3f7370bf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3f73708579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3f7374c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3f6b01d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3f6b01dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3f6b01f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3f6b00e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3f6c6526db start_thread at ??:0
@ 0x7f3f678c961f clone at ??:0
W20260927 00:45:00.306475 16912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:00.306881 16912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:00.306965 16912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:00.315250 16912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:00.315312 16912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:00.315431 16912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:00.323326 16912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33675
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:00.324388 16912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:00.325524 16912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:00.334868 16957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.334867 16958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.334861 16960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.335201 16912 server_base.cc:1034] running on GCE node
I20260927 00:45:00.335589 16912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:00.336251 16912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:00.337425 16912 hybrid_clock.cc:648] HybridClock initialized: now 1790469900337412 us; error 37 us; skew 500 ppm
Sep 27 00:45:00 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:00.340023 16912 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:45:00.340986 16912 webserver.cc:460] Webserver started at http://127.0.98.1:33675/ using document root <none> and password file <none>
I20260927 00:45:00.341522 16912 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:00.341579 16912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:00.345599 16912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:00.347723 16967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.349018 16912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:00.349138 16912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "d1d8f696be144e42af056bf67e4c69af"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "96f1082eb58387c23c230b9e9ef22286"
server_key_iv: "4264178b57b8b517c535362b7882b66c"
server_key_version: "encryptionkey@0"
I20260927 00:45:00.349622 16912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:00.367003 16912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:00.370036 16912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:00.370244 16912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:00.370862 16912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:00.372014 16974 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 00:45:00.377295 16912 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 00:45:00.377370 16912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 00:45:00.377449 16912 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 00:45:00.378998 16912 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 00:45:00.379065 16912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 00:45:00.379338 16974 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Bootstrap starting.
I20260927 00:45:00.389731 16912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41325
I20260927 00:45:00.389838 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41325 every 8 connection(s)
I20260927 00:45:00.390991 16912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:00.396782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 16912
I20260927 00:45:00.397316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36379
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=42847
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:00.399456 16974 log.cc:824] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:00.409129 16974 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 00:45:00.409559 16974 tablet_bootstrap.cc:492] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Bootstrap complete.
I20260927 00:45:00.410959 16974 ts_tablet_manager.cc:1397] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.007s
I20260927 00:45:00.414104 16974 raft_consensus.cc:348] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:00.414621 16974 raft_consensus.cc:729] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8f696be144e42af056bf67e4c69af, State: Initialized, Role: FOLLOWER
I20260927 00:45:00.415107 16974 consensus_queue.cc:260] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:00.416096 16974 ts_tablet_manager.cc:1428] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 00:45:00.429200 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:00.393221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:58683)
0927 00:45:00.393767 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.393824 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:00.395069 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:00.428404 (+ 33335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":144,"threads_started":1}
W20260927 00:45:00.429704 17082 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36079 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36079: Client connection negotiation failed: client connection to 127.0.98.62:36079: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 00:45:00.430373 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.430104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:49387)
0927 00:45:00.430262 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.430326 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 00:45:00.430979 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.430780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:40473)
0927 00:45:00.430884 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.430927 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:45:00.431136 17082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:45:00.513458 17085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:00.513736 17085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:00.513811 17085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:00.517756 17085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:00.517841 17085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:00.517935 17085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:00.522995 17085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42847
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:00.524008 17085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:00.525022 17085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:00.534493 17095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.534534 17093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.534534 17085 server_base.cc:1034] running on GCE node
W20260927 00:45:00.534561 17092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.535132 17085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:00.535732 17085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:00.536942 17085 hybrid_clock.cc:648] HybridClock initialized: now 1790469900536918 us; error 31 us; skew 500 ppm
Sep 27 00:45:00 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:00.539657 17085 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:45:00.540519 17085 webserver.cc:460] Webserver started at http://127.0.98.2:42847/ using document root <none> and password file <none>
I20260927 00:45:00.541177 17085 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:00.541255 17085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:00.545233 17085 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:00.547447 17102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.548848 17085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:45:00.549015 17085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "ffab519300974d25bbd878ac933804c1"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "0d9a4dc8168d4b7549ba8a6c40a99164"
server_key_iv: "9c2bfd14b1d7f6c6974f94fba8c56c4b"
server_key_version: "encryptionkey@0"
I20260927 00:45:00.549660 17085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:00.568228 17085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:00.571616 17085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:00.571841 17085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:00.572453 17085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:00.573370 17085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:00.573441 17085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.573514 17085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:00.573557 17085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.582572 17085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36379
I20260927 00:45:00.582639 17215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36379 every 8 connection(s)
I20260927 00:45:00.583606 17085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:00.584326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17085
I20260927 00:45:00.584766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42979
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36079
--webserver_port=39331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:00.588667 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.585781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:56083)
0927 00:45:00.587016 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.587157 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":144,"threads_started":1}
W20260927 00:45:00.589138 17216 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36079 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36079: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
I20260927 00:45:00.589870 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.589572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:54417)
0927 00:45:00.589712 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.589786 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:45:00.590583 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.590308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:36645)
0927 00:45:00.590443 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.590504 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:45:00.590780 17216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 00:45:00.695607 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:00.695919 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:00.695966 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:00.700120 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:00.700183 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:00.700240 17218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:00.705444 17218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39331
--tserver_master_addrs=127.0.98.62:36079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:00.706386 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:00.707333 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:00.716750 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:00.716681 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.716774 17218 server_base.cc:1034] running on GCE node
W20260927 00:45:00.716908 17227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:00.717348 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:00.717988 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:00.719173 17218 hybrid_clock.cc:648] HybridClock initialized: now 1790469900719150 us; error 35 us; skew 500 ppm
Sep 27 00:45:00 dist-test-slave-pkws krb5kdc[15763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:00.721721 17218 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:45:00.722627 17218 webserver.cc:460] Webserver started at http://127.0.98.3:39331/ using document root <none> and password file <none>
I20260927 00:45:00.723167 17218 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:00.723258 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:00.726992 17218 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:00.729063 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.730279 17218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:00.730410 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "202d3e80699a436bbfb831ffa108a053"
format_stamp: "Formatted at 2026-09-27 00:44:18 on dist-test-slave-pkws"
server_key: "973b645d01a7b6fc66011af4a7e4dc93"
server_key_iv: "b66d92b0ade231e61f6494e5bcefdf05"
server_key_version: "encryptionkey@0"
I20260927 00:45:00.731099 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:00.752110 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:00.755517 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:00.755724 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:00.756318 17218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:00.757225 17218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:00.757294 17218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.757366 17218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:00.757411 17218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:00.766382 17218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42979
I20260927 00:45:00.766412 17347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42979 every 8 connection(s)
I20260927 00:45:00.767362 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:00.771221 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.769487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:36049)
0927 00:45:00.769957 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.770066 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
W20260927 00:45:00.771687 17348 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36079 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36079: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
I20260927 00:45:00.771924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17218
I20260927 00:45:00.772341 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.772122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:49487)
0927 00:45:00.772232 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.772288 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 00:45:00.773079 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:00.772824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:48221)
0927 00:45:00.772964 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:00.773011 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 00:45:00.773260 17348 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 00:45:01.432094 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.431697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:44515)
0927 00:45:01.431931 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.431986 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260927 00:45:01.591603 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.591248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:50409)
0927 00:45:01.591462 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.591512 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260927 00:45:01.774130 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:01.773728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:40149)
0927 00:45:01.773988 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:01.774037 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260927 00:45:02.249243 17355 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:02.249429 17355 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:02.250494 17355 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
Sep 27 00:45:02 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 00:45:02 dist-test-slave-pkws krb5kdc[15763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 00:45:02.273221 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.250821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36379 (local address 127.0.98.1:58269)
0927 00:45:02.251226 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.251269 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:02.251353 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:02.253126 (+ 1773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:02.253186 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:02.253699 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:02.254522 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.254536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.256475 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.256479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.258181 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.258192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.259117 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.259121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.259394 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.265091 (+ 5697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:02.265117 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:02.267572 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:02.269903 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:02.269911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:02.269927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:02.270517 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:02.270921 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:02.270925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:02.270927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:02.271194 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:02.271989 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:02.271996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:02.272399 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 00:45:02.272617 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:45:02.273031 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":455,"spinlock_wait_cycles":40064,"thread_start_us":61,"threads_started":1}
I20260927 00:45:02.273303 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.250836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60367 (local address 127.0.98.3:42979)
0927 00:45:02.251187 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:02.251207 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:02.251243 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:45:02.252180 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:02.252190 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:02.252659 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:02.253027 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:02.254709 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.256321 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.258300 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.258985 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.259147 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.267583 (+ 8436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:02.267588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:02.267600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:02.267651 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:02.269745 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:02.270656 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:02.270663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:02.270668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:02.270811 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:02.271310 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:02.271314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:02.271316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:02.271840 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:02.271981 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:02.272779 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 00:45:02.273122 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260927 00:45:02.273321 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.251129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58269 (local address 127.0.98.2:36379)
0927 00:45:02.251448 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:02.251460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:02.251496 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:45:02.252224 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:02.252231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:02.252659 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:02.253027 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:02.254710 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.256321 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.258300 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.258985 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.259147 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.267757 (+ 8610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:02.267765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:02.267780 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:02.267811 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:02.269745 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:02.270656 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:02.270663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:02.270668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:02.270811 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:02.271311 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:02.271314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:02.271316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:02.271840 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:02.271981 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:02.272779 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 00:45:02.273158 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260927 00:45:02.273819 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.250644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.98.1:60367)
0927 00:45:02.250930 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.250944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:02.251048 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:02.253126 (+ 2078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:02.253133 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:02.253700 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:02.254514 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.254536 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.256475 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.256479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.258181 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:02.258192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:02.259117 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:02.259121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:02.259396 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:02.265091 (+ 5695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:02.265129 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:02.267331 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:02.269903 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:02.269911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:02.269927 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:02.270517 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:02.270921 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:02.270925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:02.270927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:02.271203 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:02.271989 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:02.271996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:02.272399 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 00:45:02.272617 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:45:02.273659 (+ 1042us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1115,"spinlock_wait_cycles":46976,"thread_start_us":88,"threads_started":1}
W20260927 00:45:02.275667 16968 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
W20260927 00:45:02.275833 16970 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
I20260927 00:45:02.275911 16970 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1d8f696be144e42af056bf67e4c69af; no voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1
I20260927 00:45:02.276124 17355 raft_consensus.cc:2726] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:45:02.433286 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.433028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:36195)
0927 00:45:02.433149 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.433201 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:45:02.592615 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.592245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:34365)
0927 00:45:02.592469 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.592519 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260927 00:45:02.775192 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:02.774829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:35677)
0927 00:45:02.775059 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:02.775111 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260927 00:45:03.434522 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.434032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:60821)
0927 00:45:03.434339 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.434390 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260927 00:45:03.593699 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.593307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:52657)
0927 00:45:03.593563 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.593616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260927 00:45:03.776289 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:03.775902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:48265)
0927 00:45:03.776153 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:03.776201 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260927 00:45:04.315828 17365 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:04.315958 17365 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:04.316339 17365 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
W20260927 00:45:04.317256 16970 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
W20260927 00:45:04.317423 16968 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
I20260927 00:45:04.317520 16968 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1d8f696be144e42af056bf67e4c69af; no voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1
I20260927 00:45:04.317673 17365 raft_consensus.cc:2726] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:45:04.435668 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.435265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:59029)
0927 00:45:04.435509 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.435571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260927 00:45:04.594758 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.594373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:37073)
0927 00:45:04.594616 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.594668 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260927 00:45:04.777395 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:04.777010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:33845)
0927 00:45:04.777249 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:04.777304 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260927 00:45:05.437034 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.436530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:44545)
0927 00:45:05.436826 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.436939 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260927 00:45:05.595891 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.595514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:51269)
0927 00:45:05.595742 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.595793 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260927 00:45:05.778445 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:05.778063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:39083)
0927 00:45:05.778303 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:05.778356 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
I20260927 00:45:06.438199 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.437800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:49349)
0927 00:45:06.438057 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.438108 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260927 00:45:06.597200 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.596753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:40939)
0927 00:45:06.597054 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.597109 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":75,"threads_started":1}
I20260927 00:45:06.767097 17374 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:06.767237 17374 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:06.767776 17374 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
W20260927 00:45:06.768529 16970 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
W20260927 00:45:06.768623 16968 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
I20260927 00:45:06.768721 16968 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1d8f696be144e42af056bf67e4c69af; no voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1
I20260927 00:45:06.768874 17374 raft_consensus.cc:2726] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:45:06.779542 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:06.779135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:48461)
0927 00:45:06.779398 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:06.779453 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260927 00:45:07.439342 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.438960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:51573)
0927 00:45:07.439193 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.439248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260927 00:45:07.598282 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.597880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:49103)
0927 00:45:07.598142 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.598194 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
I20260927 00:45:07.780619 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:07.780222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:53299)
0927 00:45:07.780476 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:07.780526 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":66,"threads_started":1}
I20260927 00:45:08.440478 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.440070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:57259)
0927 00:45:08.440326 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.440383 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260927 00:45:08.599341 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.598945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:53631)
0927 00:45:08.599192 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.599249 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260927 00:45:08.781663 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:08.781287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:52781)
0927 00:45:08.781519 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:08.781569 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260927 00:45:09.177520 17382 raft_consensus.cc:482] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:09.177654 17382 raft_consensus.cc:504] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "202d3e80699a436bbfb831ffa108a053" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "ffab519300974d25bbd878ac933804c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36379 } } peers { permanent_uuid: "d1d8f696be144e42af056bf67e4c69af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41325 } }
I20260927 00:45:09.178138 17382 leader_election.cc:288] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979), ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379)
W20260927 00:45:09.178908 16970 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 202d3e80699a436bbfb831ffa108a053 (127.0.98.3:42979): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
W20260927 00:45:09.179009 16968 leader_election.cc:341] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ffab519300974d25bbd878ac933804c1 (127.0.98.2:36379): Not found: Tablet not found: c10a05e82cae4edcbf570a50b27a1778
I20260927 00:45:09.179121 16968 leader_election.cc:302] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1d8f696be144e42af056bf67e4c69af; no voters: 202d3e80699a436bbfb831ffa108a053, ffab519300974d25bbd878ac933804c1
I20260927 00:45:09.179277 17382 raft_consensus.cc:2726] T c10a05e82cae4edcbf570a50b27a1778 P d1d8f696be144e42af056bf67e4c69af [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 00:45:09.441758 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.441264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:36723)
0927 00:45:09.441544 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.441614 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260927 00:45:09.600436 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.600040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:39073)
0927 00:45:09.600295 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.600349 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
I20260927 00:45:09.782792 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:09.782349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:57943)
0927 00:45:09.782597 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:09.782697 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260927 00:45:10.443064 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.442670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:35069)
0927 00:45:10.442912 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.442971 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260927 00:45:10.601676 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.601286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:40503)
0927 00:45:10.601524 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.601575 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260927 00:45:10.783890 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:10.783530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:33779)
0927 00:45:10.783752 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:10.783801 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260927 00:45:11.444203 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.443813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:42521)
0927 00:45:11.444049 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.444112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260927 00:45:11.602844 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.602468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:53163)
0927 00:45:11.602705 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.602755 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260927 00:45:11.785094 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:11.784634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:34357)
0927 00:45:11.784872 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:11.784994 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260927 00:45:12.445387 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.445003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:42567)
0927 00:45:12.445241 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.445293 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260927 00:45:12.603976 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.603586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:35385)
0927 00:45:12.603831 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.603884 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260927 00:45:12.786164 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:12.785775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:53873)
0927 00:45:12.786013 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:12.786078 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260927 00:45:13.446486 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.446064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:38893)
0927 00:45:13.446337 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.446393 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260927 00:45:13.605078 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.604668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:41019)
0927 00:45:13.604928 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.604986 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260927 00:45:13.787246 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:13.786844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:36741)
0927 00:45:13.787105 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:13.787156 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
I20260927 00:45:14.447713 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.447279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:40443)
0927 00:45:14.447559 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.447616 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260927 00:45:14.606127 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.605725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:51181)
0927 00:45:14.605979 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.606033 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260927 00:45:14.788341 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:14.787935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:38761)
0927 00:45:14.788196 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:14.788251 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260927 00:45:15.449229 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.448759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.1:52237)
0927 00:45:15.449076 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.449137 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":73,"threads_started":1}
I20260927 00:45:15.607479 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.607078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.2:41315)
0927 00:45:15.607326 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.607387 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260927 00:45:15.784626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 16912
I20260927 00:45:15.789698 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:15.789304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36079 (local address 127.0.98.3:59463)
0927 00:45:15.789553 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:15.789611 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260927 00:45:15.790370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17085
I20260927 00:45:15.797317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17218
2026-09-27T00:45:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59707 ms)
[----------] 30 tests from SecurityITest (237672 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:16 dist-test-slave-pkws krb5kdc[17406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:16 dist-test-slave-pkws krb5kdc[17406](info): set up 2 sockets
Sep 27 00:45:16 dist-test-slave-pkws krb5kdc[17406](info): commencing operation
krb5kdc: starting...
W20260927 00:45:17.280589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:17 dist-test-slave-pkws krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:17.403388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34989
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:17.513058 17419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:17.513295 17419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:17.513334 17419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:17.517180 17419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:17.517225 17419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:17.517244 17419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:17.517261 17419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:17.517277 17419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:17.522212 17419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:17.523121 17419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:17.524029 17419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:17.532462 17424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.532503 17425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.532564 17419 server_base.cc:1034] running on GCE node
W20260927 00:45:17.532475 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.533316 17419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:17.534348 17419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:17.535523 17419 hybrid_clock.cc:648] HybridClock initialized: now 1790469917535494 us; error 41 us; skew 500 ppm
Sep 27 00:45:17 dist-test-slave-pkws krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:17.538363 17419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:17.539280 17419 webserver.cc:460] Webserver started at http://127.0.98.62:42213/ using document root <none> and password file <none>
I20260927 00:45:17.539825 17419 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:17.539884 17419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:17.540119 17419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:17.542145 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "dfdc9f2b6d604d78a7c64feacb83252b"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "83c5a1c66dc4501af295f362b01a1efc"
server_key_iv: "d5dbe88b9ea73d3c6087f3d07f455848"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.542680 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfdc9f2b6d604d78a7c64feacb83252b"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "83c5a1c66dc4501af295f362b01a1efc"
server_key_iv: "d5dbe88b9ea73d3c6087f3d07f455848"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.546200 17419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:17.548410 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.549707 17419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:17.549851 17419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "dfdc9f2b6d604d78a7c64feacb83252b"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "83c5a1c66dc4501af295f362b01a1efc"
server_key_iv: "d5dbe88b9ea73d3c6087f3d07f455848"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.549960 17419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:17.569837 17419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:17.572564 17419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:17.572762 17419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:17.579463 17419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34989
I20260927 00:45:17.579488 17486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34989 every 8 connection(s)
I20260927 00:45:17.580341 17419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:17.583003 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:17.588464 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: Bootstrap starting.
I20260927 00:45:17.589571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17419
I20260927 00:45:17.589722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:17.590027 392 external_mini_cluster.cc:1428] Setting key a9ef8bec47ee7a30d8bfd9489a3034d6
I20260927 00:45:17.591291 17487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:17.591993 17487 log.cc:824] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:17.593839 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: No bootstrap required, opened a new log
I20260927 00:45:17.596709 17487 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } }
I20260927 00:45:17.596967 17487 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:17.597048 17487 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdc9f2b6d604d78a7c64feacb83252b, State: Initialized, Role: FOLLOWER
I20260927 00:45:17.597543 17487 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } }
I20260927 00:45:17.597714 17487 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:17.597805 17487 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:17.597908 17487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:17.598912 17487 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } }
Sep 27 00:45:17 dist-test-slave-pkws krb5kdc[17406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:17.599270 17487 leader_election.cc:302] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfdc9f2b6d604d78a7c64feacb83252b; no voters:
I20260927 00:45:17.599629 17487 leader_election.cc:288] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:17.599741 17492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:17.600008 17492 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [term 1 LEADER]: Becoming Leader. State: Replica: dfdc9f2b6d604d78a7c64feacb83252b, State: Running, Role: LEADER
I20260927 00:45:17.600394 17492 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } }
I20260927 00:45:17.601055 17487 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:17.602726 17493 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } } }
I20260927 00:45:17.602867 17493 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:17.603866 17494 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfdc9f2b6d604d78a7c64feacb83252b. Latest consensus state: current_term: 1 leader_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdc9f2b6d604d78a7c64feacb83252b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34989 } } }
I20260927 00:45:17.603978 17494 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:17.605691 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.591087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.98.62:34989)
0927 00:45:17.591579 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:17.591594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:17.591627 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:17.592273 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:17.592290 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:17.592712 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:17.593082 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:17.593869 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.594845 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.595685 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.596586 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.596759 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.599794 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:17.599802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:17.599820 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:17.599862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:17.602107 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.602684 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.602692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.602701 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.602885 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.603278 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.603281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.603283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.603771 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:17.603976 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:17.604432 (+ 456us) server_negotiation.cc:299] Negotiation successful
0927 00:45:17.604748 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":131,"threads_started":1}
W20260927 00:45:17.606055 17506 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:17.606142 17506 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:17.606228 17507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:17.607033 17507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:17.612289 17507 catalog_manager.cc:1347] Generated new cluster ID: 3facf7684077420c94a598cee195f46b
I20260927 00:45:17.612361 17507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:17.641597 17507 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:17.642520 17507 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:17.651235 17507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: Generated new TSK 0
I20260927 00:45:17.651872 17507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:45:17.684073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:17.789777 17513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:17.789995 17513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:17.790062 17513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:17.793992 17513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:17.794042 17513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:17.794126 17513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:17.799096 17513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:17.800043 17513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:17.801028 17513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:17.810170 17519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:17.810186 17518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.810235 17513 server_base.cc:1034] running on GCE node
W20260927 00:45:17.810170 17521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:17.810745 17513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:17.811352 17513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:17.812512 17513 hybrid_clock.cc:648] HybridClock initialized: now 1790469917812498 us; error 25 us; skew 500 ppm
Sep 27 00:45:17 dist-test-slave-pkws krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:17.815316 17513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:45:17.816044 17513 webserver.cc:460] Webserver started at http://127.0.98.1:40659/ using document root <none> and password file <none>
I20260927 00:45:17.816545 17513 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:17.816602 17513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:17.816753 17513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:17.818720 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "f96d492c02244e9f9cfecc769a273480"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "a6bfaaf4242d7869badb7b78e6b4e281"
server_key_iv: "e0e4cfafb98ac977c75d41aff9fd3dfe"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.819229 17513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f96d492c02244e9f9cfecc769a273480"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "a6bfaaf4242d7869badb7b78e6b4e281"
server_key_iv: "e0e4cfafb98ac977c75d41aff9fd3dfe"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.822686 17513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:17.827582 17528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.828827 17513 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 00:45:17.828992 17513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f96d492c02244e9f9cfecc769a273480"
format_stamp: "Formatted at 2026-09-27 00:45:17 on dist-test-slave-pkws"
server_key: "a6bfaaf4242d7869badb7b78e6b4e281"
server_key_iv: "e0e4cfafb98ac977c75d41aff9fd3dfe"
server_key_version: "encryptionkey@0"
I20260927 00:45:17.829121 17513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:17.848282 17513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:17.851431 17513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:17.851625 17513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:17.852171 17513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:17.853021 17513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:17.853087 17513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.853163 17513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:17.853206 17513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:17.861667 17513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46017
I20260927 00:45:17.861711 17641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46017 every 8 connection(s)
I20260927 00:45:17.862517 17513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:17.870621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17513
I20260927 00:45:17.870767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:17.871093 392 external_mini_cluster.cc:1428] Setting key 8c9580de0e07524390f15152cc9ec8ab
Sep 27 00:45:17 dist-test-slave-pkws krb5kdc[17406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:17.875545 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.864189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34967 (local address 127.0.98.62:34989)
0927 00:45:17.864323 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:17.864327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:17.865126 (+ 799us) server_negotiation.cc:407] Connection header received
0927 00:45:17.866278 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:17.866281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:17.866324 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:17.866399 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:17.867442 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.867954 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.869066 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.869304 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.869384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.871723 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:17.871727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:17.871730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:17.871756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:17.873453 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.873935 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.873941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.873946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.874039 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:17.874483 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:17.874486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:17.874487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:17.874649 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:17.874723 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:17.875320 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 00:45:17.875412 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:45:17.876662 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:17.864524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34989 (local address 127.0.98.1:34967)
0927 00:45:17.864990 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:17.865030 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:17.866097 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:17.866560 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:17.866566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:17.866880 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:17.867321 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.867334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.868058 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.868061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.868961 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:17.868976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:17.869424 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:17.869427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:17.869568 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:17.870017 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:17.870031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:17.871567 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:17.873575 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.873579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.873589 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.873836 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.874145 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:17.874150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:17.874154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:17.874379 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:17.874760 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:17.874769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:17.875067 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 00:45:17.875285 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 00:45:17.875620 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260927 00:45:17.877848 17642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34989
I20260927 00:45:17.878090 17642 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:17.878536 17642 heartbeater.cc:511] Master 127.0.98.62:34989 requested a full tablet report, sending...
I20260927 00:45:17.879945 17451 ts_manager.cc:194] Registered new tserver with Master: f96d492c02244e9f9cfecc769a273480 (127.0.98.1:46017)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:45:17.881716 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:45:17.899442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:18.004822 17647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:18.005113 17647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:18.005191 17647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:18.009423 17647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:18.009496 17647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:18.009580 17647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:18.014707 17647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:18.015652 17647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:18.016652 17647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:18.026018 17647 server_base.cc:1034] running on GCE node
W20260927 00:45:18.026046 17652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.025961 17653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.025940 17655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.026556 17647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:18.027185 17647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:18.028373 17647 hybrid_clock.cc:648] HybridClock initialized: now 1790469918028344 us; error 39 us; skew 500 ppm
Sep 27 00:45:18 dist-test-slave-pkws krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:18.031133 17647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:45:18.031968 17647 webserver.cc:460] Webserver started at http://127.0.98.2:38655/ using document root <none> and password file <none>
I20260927 00:45:18.032510 17647 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:18.032598 17647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:18.032812 17647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:18.035005 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "38b8df53f6484533a3d0da7ce01de0c0"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "039caff788055049105332cec1660e11"
server_key_iv: "52df02c5694003340ccfcb49aeb11dd5"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.035568 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38b8df53f6484533a3d0da7ce01de0c0"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "039caff788055049105332cec1660e11"
server_key_iv: "52df02c5694003340ccfcb49aeb11dd5"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.039124 17647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:45:18.041420 17662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.042570 17647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:18.042701 17647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "38b8df53f6484533a3d0da7ce01de0c0"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "039caff788055049105332cec1660e11"
server_key_iv: "52df02c5694003340ccfcb49aeb11dd5"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.042805 17647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:18.064442 17647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:18.067631 17647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:18.067826 17647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:18.068379 17647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:18.069213 17647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:18.069284 17647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.069355 17647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:18.069396 17647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.078128 17647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35517
I20260927 00:45:18.078168 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35517 every 8 connection(s)
I20260927 00:45:18.078964 17647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:18.085170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17647
I20260927 00:45:18.085278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:18.085564 392 external_mini_cluster.cc:1428] Setting key 29b685dda22f7a633a7918e4eb4c243b
Sep 27 00:45:18 dist-test-slave-pkws krb5kdc[17406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:18.092195 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.080603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53575 (local address 127.0.98.62:34989)
0927 00:45:18.080748 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.080751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.081548 (+ 797us) server_negotiation.cc:407] Connection header received
0927 00:45:18.082793 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.082796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.082855 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.082958 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.084093 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.084609 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.085909 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.086131 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.086244 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.088652 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.088656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.088658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:18.088701 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.090202 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.090696 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.090701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.090703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.090782 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.091123 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.091128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.091130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.091313 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.091418 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.091894 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.092016 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:45:18.093061 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.080968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34989 (local address 127.0.98.2:53575)
0927 00:45:18.081396 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.081436 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.082571 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.083131 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.083141 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.083468 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:18.083916 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.083930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.084741 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.084745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.085784 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.085794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.086291 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.086295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.086454 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.087004 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:18.087019 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:18.088489 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:18.090321 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.090326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.090339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.090583 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.090876 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.090881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.090883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.091023 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.091432 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:18.091439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:18.091669 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.091850 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.092147 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260927 00:45:18.094165 17776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34989
I20260927 00:45:18.094424 17776 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:18.094982 17776 heartbeater.cc:511] Master 127.0.98.62:34989 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:45:18.096014 17451 ts_manager.cc:194] Registered new tserver with Master: 38b8df53f6484533a3d0da7ce01de0c0 (127.0.98.2:35517)
I20260927 00:45:18.096692 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:45:18.114115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:18.221241 17781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:18.221462 17781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:18.221503 17781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:18.225643 17781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:18.225696 17781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:18.225751 17781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:18.230744 17781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:18.231571 17781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:18.232426 17781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:18.241637 17787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.241685 17786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.241647 17789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.241724 17781 server_base.cc:1034] running on GCE node
I20260927 00:45:18.242213 17781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:18.242820 17781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:18.244007 17781 hybrid_clock.cc:648] HybridClock initialized: now 1790469918243974 us; error 44 us; skew 500 ppm
Sep 27 00:45:18 dist-test-slave-pkws krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:18.246786 17781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:45:18.247567 17781 webserver.cc:460] Webserver started at http://127.0.98.3:43133/ using document root <none> and password file <none>
I20260927 00:45:18.248108 17781 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:18.248198 17781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:18.248415 17781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:18.250384 17781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "f529864600a24079bb48353c79f350dd"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "90b5a99faa176d2c2d3717e2ea7f0b10"
server_key_iv: "3e1d35890816d65c2a8b151fc83b7a24"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.250947 17781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f529864600a24079bb48353c79f350dd"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "90b5a99faa176d2c2d3717e2ea7f0b10"
server_key_iv: "3e1d35890816d65c2a8b151fc83b7a24"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.254622 17781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 00:45:18.256969 17796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.258147 17781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:18.258284 17781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "f529864600a24079bb48353c79f350dd"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "90b5a99faa176d2c2d3717e2ea7f0b10"
server_key_iv: "3e1d35890816d65c2a8b151fc83b7a24"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.258394 17781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:18.274339 17781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:18.277145 17781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:18.277335 17781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:18.277889 17781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:18.278671 17781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:18.278739 17781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.278810 17781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:18.278864 17781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.287429 17781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46401
I20260927 00:45:18.287525 17909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46401 every 8 connection(s)
I20260927 00:45:18.288542 17781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:18.289636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17781
I20260927 00:45:18.289776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:18.290591 392 external_mini_cluster.cc:1428] Setting key ba9f83b5803d4706071d3dc8c055213a
Sep 27 00:45:18 dist-test-slave-pkws krb5kdc[17406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:18.301419 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.290982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49109 (local address 127.0.98.62:34989)
0927 00:45:18.291119 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.291123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.291658 (+ 535us) server_negotiation.cc:407] Connection header received
0927 00:45:18.292949 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.292952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.293007 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.293072 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.294130 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.294640 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.295704 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.295906 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.295980 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.298361 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.298364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.298366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:18.298385 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.299612 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.300041 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.300044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.300045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.300120 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.300398 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.300400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.300401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.300581 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.300655 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.301154 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.301244 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:45:18.302192 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.291002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34989 (local address 127.0.98.3:49109)
0927 00:45:18.291503 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.291555 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.292721 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.293205 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.293212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.293552 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:18.294012 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.294024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.294742 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.294744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.295595 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.295604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.296025 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.296028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.296164 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.296767 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:18.296783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:18.298226 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:18.299711 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.299715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.299725 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.299954 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.300198 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:18.300201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:18.300202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:18.300323 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:18.300674 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:18.300679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:18.300964 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.301117 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.301349 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":167,"threads_started":1}
I20260927 00:45:18.303454 17910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34989
I20260927 00:45:18.303689 17910 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:18.304237 17910 heartbeater.cc:511] Master 127.0.98.62:34989 requested a full tablet report, sending...
I20260927 00:45:18.305219 17451 ts_manager.cc:194] Registered new tserver with Master: f529864600a24079bb48353c79f350dd (127.0.98.3:46401)
I20260927 00:45:18.305779 17451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49109
I20260927 00:45:18.315691 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:18.324075 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.317561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.98.62:34989)
0927 00:45:18.317651 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.317654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.317866 (+ 212us) server_negotiation.cc:407] Connection header received
0927 00:45:18.317954 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.317956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.317994 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.318054 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.318645 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.319123 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.319882 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.320079 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.320160 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.321429 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.321432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.321434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:18.321455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.322661 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.323029 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.323031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.323033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.323100 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:18.323323 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:18.323325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:18.323326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:18.323513 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.323612 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.323786 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.323868 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 00:45:18.326934 17451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:18.329957 17451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:18.343707 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35517 (local address 127.0.0.1:49978)
0927 00:45:18.339189 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.339202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.339280 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.339997 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.340000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.340019 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.340102 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.340107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.341311 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.341314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.342317 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.342327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.343123 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.343126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.343242 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.343277 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.343435 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.343547 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260927 00:45:18.343712 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.98.2:35517)
0927 00:45:18.339594 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.339609 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.339633 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:45:18.339713 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.339717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.339861 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.339971 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.340258 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.341185 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.342454 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.342999 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.343124 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.343269 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.343466 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.343553 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":103,"threads_started":1}
I20260927 00:45:18.344065 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.3:46401)
0927 00:45:18.338822 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.338834 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.338875 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:45:18.338996 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.339001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.339231 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.339399 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.339666 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.340727 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.342380 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.343246 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.343326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.343463 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.343809 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.343922 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":108,"threads_started":1}
I20260927 00:45:18.344489 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.337867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46401 (local address 127.0.0.1:52276)
0927 00:45:18.338635 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.338680 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.338802 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.339399 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.339405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.339427 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.339553 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.339560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.340876 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.340879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.342251 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.342261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.343555 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.343559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.343635 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.343656 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.344289 (+ 633us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.344366 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":74,"threads_started":1}
I20260927 00:45:18.345261 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46017 (local address 127.0.0.1:57734)
0927 00:45:18.340155 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.340193 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.340286 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.340690 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.340693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.340711 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.340795 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.340801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.342204 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.342208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.343450 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.343460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.344855 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.344859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.344976 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.345003 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.345073 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.345145 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1993,"thread_start_us":97,"threads_started":1}
I20260927 00:45:18.345717 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.338658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.98.1:46017)
0927 00:45:18.339048 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.339062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.340305 (+ 1243us) server_negotiation.cc:407] Connection header received
0927 00:45:18.340383 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.340387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.340550 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.340692 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.340911 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.342053 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.343575 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.344540 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.344655 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.344794 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.345418 (+ 624us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.345509 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":84,"threads_started":1}
I20260927 00:45:18.346917 17710 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c (DEFAULT_TABLE table=test-table [id=ed5c939e5d8543aa90c90a0bb8b96afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.346984 17844 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c (DEFAULT_TABLE table=test-table [id=ed5c939e5d8543aa90c90a0bb8b96afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.347779 17844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6e435e58d45969f8f1c1ca73bba3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.347780 17710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6e435e58d45969f8f1c1ca73bba3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.351476 17576 tablet_service.cc:1467] Processing CreateTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c (DEFAULT_TABLE table=test-table [id=ed5c939e5d8543aa90c90a0bb8b96afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:18.352440 17576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0b6e435e58d45969f8f1c1ca73bba3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.354888 17927 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Bootstrap starting.
I20260927 00:45:18.357170 17927 tablet_bootstrap.cc:654] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.357182 17928 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Bootstrap starting.
I20260927 00:45:18.358477 17927 log.cc:824] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.358883 17928 tablet_bootstrap.cc:654] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.359203 17930 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: Bootstrap starting.
I20260927 00:45:18.360069 17928 log.cc:824] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.361183 17930 tablet_bootstrap.cc:654] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.361496 17927 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: No bootstrap required, opened a new log
I20260927 00:45:18.361720 17927 ts_tablet_manager.cc:1397] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 00:45:18.361954 17928 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: No bootstrap required, opened a new log
I20260927 00:45:18.362156 17928 ts_tablet_manager.cc:1397] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:18.362362 17930 log.cc:824] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.364279 17930 tablet_bootstrap.cc:492] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: No bootstrap required, opened a new log
I20260927 00:45:18.364462 17930 ts_tablet_manager.cc:1397] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:18.365415 17928 raft_consensus.cc:348] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.365653 17928 raft_consensus.cc:374] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.365491 17927 raft_consensus.cc:348] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.365708 17928 raft_consensus.cc:729] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8df53f6484533a3d0da7ce01de0c0, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.365731 17927 raft_consensus.cc:374] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.365774 17927 raft_consensus.cc:729] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f529864600a24079bb48353c79f350dd, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.366142 17928 consensus_queue.cc:260] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.366354 17927 consensus_queue.cc:260] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.367048 17928 ts_tablet_manager.cc:1428] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:45:18.367303 17927 ts_tablet_manager.cc:1428] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:45:18.367399 17776 heartbeater.cc:503] Master 127.0.98.62:34989 was elected leader, sending a full tablet report...
I20260927 00:45:18.367630 17910 heartbeater.cc:503] Master 127.0.98.62:34989 was elected leader, sending a full tablet report...
I20260927 00:45:18.368021 17930 raft_consensus.cc:348] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.368178 17930 raft_consensus.cc:374] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.368213 17930 raft_consensus.cc:729] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96d492c02244e9f9cfecc769a273480, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.368728 17930 consensus_queue.cc:260] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.369438 17642 heartbeater.cc:503] Master 127.0.98.62:34989 was elected leader, sending a full tablet report...
I20260927 00:45:18.369643 17930 ts_tablet_manager.cc:1428] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:45:18.376443 17934 raft_consensus.cc:482] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:18.376590 17934 raft_consensus.cc:504] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.377585 17934 leader_election.cc:288] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38b8df53f6484533a3d0da7ce01de0c0 (127.0.98.2:35517), f96d492c02244e9f9cfecc769a273480 (127.0.98.1:46017)
I20260927 00:45:18.382589 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.378034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59971 (local address 127.0.98.1:46017)
0927 00:45:18.378155 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.378161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.378288 (+ 127us) server_negotiation.cc:407] Connection header received
0927 00:45:18.378393 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.378394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.378424 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.378479 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.378752 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.379454 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.381332 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.381924 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.382037 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.382096 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.382401 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.382470 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:45:18.382589 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.377853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47199 (local address 127.0.98.2:35517)
0927 00:45:18.377977 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.377981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.377994 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:18.378075 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.378077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.378119 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.378189 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:18.378617 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.379407 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.381332 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.381939 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.382037 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.382096 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.382401 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.382474 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:18.382668 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.377749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35517 (local address 127.0.98.3:47199)
0927 00:45:18.377862 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.377900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.377980 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.378223 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.378229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.378250 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.378434 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.378440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.379542 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.379545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.381207 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.381217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.382075 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.382079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.382236 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.382286 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.382370 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.382523 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":277,"spinlock_wait_cycles":19968}
I20260927 00:45:18.382681 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.377851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46017 (local address 127.0.98.3:59971)
0927 00:45:18.378191 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.378204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.378303 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.378519 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.378524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.378543 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:18.378657 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.378660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.379547 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.379549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.381206 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.381217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.382075 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.382079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.382230 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.382289 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.382370 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.382510 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":265,"spinlock_wait_cycles":896,"thread_start_us":93,"threads_started":1}
I20260927 00:45:18.383242 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b6e435e58d45969f8f1c1ca73bba3c" candidate_uuid: "f529864600a24079bb48353c79f350dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96d492c02244e9f9cfecc769a273480" is_pre_election: true
I20260927 00:45:18.383242 17730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b6e435e58d45969f8f1c1ca73bba3c" candidate_uuid: "f529864600a24079bb48353c79f350dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8df53f6484533a3d0da7ce01de0c0" is_pre_election: true
I20260927 00:45:18.383522 17730 raft_consensus.cc:2454] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f529864600a24079bb48353c79f350dd in term 0.
I20260927 00:45:18.383522 17596 raft_consensus.cc:2454] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f529864600a24079bb48353c79f350dd in term 0.
I20260927 00:45:18.383955 17800 leader_election.cc:302] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f529864600a24079bb48353c79f350dd, f96d492c02244e9f9cfecc769a273480; no voters:
I20260927 00:45:18.384207 17934 raft_consensus.cc:2781] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:18.384298 17934 raft_consensus.cc:482] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:18.384388 17934 raft_consensus.cc:3037] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.385530 17934 raft_consensus.cc:504] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.385854 17934 leader_election.cc:288] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [CANDIDATE]: Term 1 election: Requested vote from peers 38b8df53f6484533a3d0da7ce01de0c0 (127.0.98.2:35517), f96d492c02244e9f9cfecc769a273480 (127.0.98.1:46017)
I20260927 00:45:18.386209 17596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b6e435e58d45969f8f1c1ca73bba3c" candidate_uuid: "f529864600a24079bb48353c79f350dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96d492c02244e9f9cfecc769a273480"
I20260927 00:45:18.386206 17730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0b6e435e58d45969f8f1c1ca73bba3c" candidate_uuid: "f529864600a24079bb48353c79f350dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8df53f6484533a3d0da7ce01de0c0"
I20260927 00:45:18.386322 17730 raft_consensus.cc:3037] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.386322 17596 raft_consensus.cc:3037] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.387506 17596 raft_consensus.cc:2454] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f529864600a24079bb48353c79f350dd in term 1.
I20260927 00:45:18.387523 17730 raft_consensus.cc:2454] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f529864600a24079bb48353c79f350dd in term 1.
I20260927 00:45:18.387823 17797 leader_election.cc:302] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38b8df53f6484533a3d0da7ce01de0c0, f529864600a24079bb48353c79f350dd, f96d492c02244e9f9cfecc769a273480; no voters:
I20260927 00:45:18.388036 17934 raft_consensus.cc:2781] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:18.388281 17934 raft_consensus.cc:686] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 1 LEADER]: Becoming Leader. State: Replica: f529864600a24079bb48353c79f350dd, State: Running, Role: LEADER
I20260927 00:45:18.388581 17934 consensus_queue.cc:237] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } }
I20260927 00:45:18.391526 17450 catalog_manager.cc:5495] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd reported cstate change: term changed from 0 to 1, leader changed from <none> to f529864600a24079bb48353c79f350dd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f529864600a24079bb48353c79f350dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f529864600a24079bb48353c79f350dd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 } health_report { overall_health: UNKNOWN } } }
I20260927 00:45:18.404942 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.401714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.3:46401)
0927 00:45:18.401810 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.401814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.401904 (+ 90us) server_negotiation.cc:407] Connection header received
0927 00:45:18.402026 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.402028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.402090 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.402169 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:18.402378 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.402867 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.403709 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.403907 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.403984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.404211 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:18.404580 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:18.404670 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.404725 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.404766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 00:45:18.410254 17730 raft_consensus.cc:1261] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 1 FOLLOWER]: Refusing update from remote peer f529864600a24079bb48353c79f350dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:18.410255 17596 raft_consensus.cc:1261] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 1 FOLLOWER]: Refusing update from remote peer f529864600a24079bb48353c79f350dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:18.410904 17937 consensus_queue.cc:1035] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8df53f6484533a3d0da7ce01de0c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:18.411044 17934 consensus_queue.cc:1035] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96d492c02244e9f9cfecc769a273480" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:18.418613 17941 mvcc.cc:205] Tried to move back new op lower bound from 7333764785801719808 to 7333764785722437632. Current Snapshot: MvccSnapshot[applied={T|T < 7333764785801719808}]
I20260927 00:45:18.427119 17450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:18.427276 17450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:18.429499 17450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfdc9f2b6d604d78a7c64feacb83252b: Sending DeleteTablet for 3 replicas of tablet c0b6e435e58d45969f8f1c1ca73bba3c
I20260927 00:45:18.430214 17844 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52276
I20260927 00:45:18.430284 17710 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49978
I20260927 00:45:18.430346 17576 tablet_service.cc:1514] Processing DeleteTablet for tablet c0b6e435e58d45969f8f1c1ca73bba3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57734
I20260927 00:45:18.430742 17952 tablet_replica.cc:330] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480: stopping tablet replica
I20260927 00:45:18.430757 17950 tablet_replica.cc:330] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: stopping tablet replica
I20260927 00:45:18.430971 17951 tablet_replica.cc:330] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: stopping tablet replica
I20260927 00:45:18.431031 17952 raft_consensus.cc:2229] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:18.431228 17951 raft_consensus.cc:2229] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:18.431294 17952 raft_consensus.cc:2258] T c0b6e435e58d45969f8f1c1ca73bba3c P f96d492c02244e9f9cfecc769a273480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.431490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17513
I20260927 00:45:18.431492 17951 raft_consensus.cc:2258] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.433238 17951 ts_tablet_manager.cc:1905] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:18.433960 17950 raft_consensus.cc:2229] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:18.434356 17950 raft_consensus.cc:2258] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:18.435587 17950 ts_tablet_manager.cc:1905] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:18.436084 17951 ts_tablet_manager.cc:1918] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:18.436179 17951 log.cc:1196] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/c0b6e435e58d45969f8f1c1ca73bba3c
I20260927 00:45:18.436518 17951 ts_tablet_manager.cc:1939] T c0b6e435e58d45969f8f1c1ca73bba3c P 38b8df53f6484533a3d0da7ce01de0c0: Deleting consensus metadata
I20260927 00:45:18.437660 17435 catalog_manager.cc:4841] TS 38b8df53f6484533a3d0da7ce01de0c0 (127.0.98.2:35517): tablet c0b6e435e58d45969f8f1c1ca73bba3c (table test-table [id=ed5c939e5d8543aa90c90a0bb8b96afd]) successfully deleted
W20260927 00:45:18.439479 17438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46017 (error 108)
I20260927 00:45:18.439606 17950 ts_tablet_manager.cc:1918] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:18.439693 17950 log.cc:1196] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/c0b6e435e58d45969f8f1c1ca73bba3c
I20260927 00:45:18.440033 17950 ts_tablet_manager.cc:1939] T c0b6e435e58d45969f8f1c1ca73bba3c P f529864600a24079bb48353c79f350dd: Deleting consensus metadata
I20260927 00:45:18.440280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17647
I20260927 00:45:18.441820 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.441526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46017 (local address 127.0.0.1:57742)
0927 00:45:18.441678 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.441740 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 00:45:18.442824 17436 catalog_manager.cc:4841] TS f529864600a24079bb48353c79f350dd (127.0.98.3:46401): tablet c0b6e435e58d45969f8f1c1ca73bba3c (table test-table [id=ed5c939e5d8543aa90c90a0bb8b96afd]) successfully deleted
W20260927 00:45:18.445027 17438 catalog_manager.cc:4584] TS f96d492c02244e9f9cfecc769a273480 (127.0.98.1:46017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0b6e435e58d45969f8f1c1ca73bba3c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46017: connect: Connection refused (error 111)
I20260927 00:45:18.447201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17781
I20260927 00:45:18.453219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17419
2026-09-27T00:45:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T00:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:18Z Disabled control of system clock
I20260927 00:45:18.482427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40333
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:18.588647 17957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:18.588909 17957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:18.588945 17957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:18.592782 17957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:18.592828 17957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:18.592844 17957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:18.592859 17957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:18.592875 17957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:18.597946 17957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:18.598860 17957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:18.600099 17957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:18.608321 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.608349 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.608349 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.608400 17957 server_base.cc:1034] running on GCE node
I20260927 00:45:18.609195 17957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:18.610217 17957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:18.611416 17957 hybrid_clock.cc:648] HybridClock initialized: now 1790469918611351 us; error 44 us; skew 500 ppm
I20260927 00:45:18.613304 17957 webserver.cc:460] Webserver started at http://127.0.98.62:35003/ using document root <none> and password file <none>
I20260927 00:45:18.613852 17957 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:18.613916 17957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:18.614122 17957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:18.616083 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "26fbf3bb28ad433faf3faa16bf66c75c"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "21e2ebc611ad9ffc7f98ea95429270ef"
server_key_iv: "fa30cadd5c61fdf2696ffbe2cc71a7b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.617122 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "26fbf3bb28ad433faf3faa16bf66c75c"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "21e2ebc611ad9ffc7f98ea95429270ef"
server_key_iv: "fa30cadd5c61fdf2696ffbe2cc71a7b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.620667 17957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:45:18.623018 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.624155 17957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:18.624284 17957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "26fbf3bb28ad433faf3faa16bf66c75c"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "21e2ebc611ad9ffc7f98ea95429270ef"
server_key_iv: "fa30cadd5c61fdf2696ffbe2cc71a7b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.624387 17957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:18.657218 17957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:18.657953 17957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:18.658149 17957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:18.664965 17957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40333
I20260927 00:45:18.665129 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40333 every 8 connection(s)
I20260927 00:45:18.665908 17957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:18.667578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 17957
I20260927 00:45:18.667717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:18.668004 392 external_mini_cluster.cc:1428] Setting key 0bc8c1ec3b87b5d655b2c0bf68b85ac5
I20260927 00:45:18.669142 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:18.674399 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: Bootstrap starting.
I20260927 00:45:18.676697 18024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:18.676925 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.669348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.62:40333)
0927 00:45:18.669948 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.669963 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.669996 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:18.670635 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.670651 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.671016 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.671344 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.671754 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.672682 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.673647 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.674167 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.674335 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.675222 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.675227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.675238 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:18.675594 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.675655 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:18.675675 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.675782 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.675934 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.676174 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260927 00:45:18.677532 18024 log.cc:824] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:18.679380 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: No bootstrap required, opened a new log
I20260927 00:45:18.682761 18024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } }
I20260927 00:45:18.683023 18024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:18.683095 18024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fbf3bb28ad433faf3faa16bf66c75c, State: Initialized, Role: FOLLOWER
I20260927 00:45:18.683432 18024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } }
I20260927 00:45:18.683542 18024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:18.683585 18024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:18.683651 18024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:18.684405 18024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } }
I20260927 00:45:18.684674 18024 leader_election.cc:302] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26fbf3bb28ad433faf3faa16bf66c75c; no voters:
I20260927 00:45:18.684948 18024 leader_election.cc:288] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:18.685053 18029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:18.685297 18029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [term 1 LEADER]: Becoming Leader. State: Replica: 26fbf3bb28ad433faf3faa16bf66c75c, State: Running, Role: LEADER
I20260927 00:45:18.685581 18029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } }
I20260927 00:45:18.686100 18024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:18.687384 18031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26fbf3bb28ad433faf3faa16bf66c75c. Latest consensus state: current_term: 1 leader_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } } }
I20260927 00:45:18.687536 18031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:18.687386 18030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fbf3bb28ad433faf3faa16bf66c75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40333 } } }
I20260927 00:45:18.687876 18030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:18.688473 18038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:18.691469 18038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:18.696750 18038 catalog_manager.cc:1347] Generated new cluster ID: 4ce06029682b4b3a984194917c3a5afb
I20260927 00:45:18.696825 18038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:18.710618 18038 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:18.711601 18038 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:18.718036 18038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: Generated new TSK 0
I20260927 00:45:18.718616 18038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:18.722146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:18.831770 18048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:18.832017 18048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:18.832055 18048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:18.836020 18048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:18.836081 18048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:18.836138 18048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:18.840830 18048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:18.841953 18048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:18.843302 18048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:18.852622 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.852669 18048 server_base.cc:1034] running on GCE node
W20260927 00:45:18.852650 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:18.852603 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:18.853194 18048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:18.853814 18048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:18.854995 18048 hybrid_clock.cc:648] HybridClock initialized: now 1790469918854981 us; error 25 us; skew 500 ppm
I20260927 00:45:18.856761 18048 webserver.cc:460] Webserver started at http://127.0.98.1:36751/ using document root <none> and password file <none>
I20260927 00:45:18.857372 18048 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:18.857486 18048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:18.857728 18048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:18.859690 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ecd347926d1456faf2adb81cad2b0ba"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "1c5f622212f4ccfd09112425b8253e03"
server_key_iv: "4eba2578b24610cb40eb00852dbf7c4e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.860245 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ecd347926d1456faf2adb81cad2b0ba"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "1c5f622212f4ccfd09112425b8253e03"
server_key_iv: "4eba2578b24610cb40eb00852dbf7c4e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.863801 18048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:18.866202 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.867363 18048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:18.867497 18048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "8ecd347926d1456faf2adb81cad2b0ba"
format_stamp: "Formatted at 2026-09-27 00:45:18 on dist-test-slave-pkws"
server_key: "1c5f622212f4ccfd09112425b8253e03"
server_key_iv: "4eba2578b24610cb40eb00852dbf7c4e"
server_key_version: "encryptionkey@0"
I20260927 00:45:18.867609 18048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:18.884128 18048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:18.884750 18048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:18.884992 18048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:18.885636 18048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:18.886662 18048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:18.886721 18048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.886783 18048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:18.886813 18048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:18.895795 18048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41597
I20260927 00:45:18.895821 18175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41597 every 8 connection(s)
I20260927 00:45:18.896654 18048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:18.898236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18048
I20260927 00:45:18.898372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:18.898701 392 external_mini_cluster.cc:1428] Setting key 3675480838dee6d7233b0e0f920f1429
I20260927 00:45:18.900620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:18.906848 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.898405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48763 (local address 127.0.98.62:40333)
0927 00:45:18.898584 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:18.898590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:18.899887 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 00:45:18.901401 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:18.901404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:18.901460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:18.901560 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:18.902400 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.902941 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.904706 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.905022 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.905108 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.906377 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:18.906380 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:18.906382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:18.906393 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:18.906411 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:18.906419 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:18.906497 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:18.906679 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:45:18.906738 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 00:45:18.908068 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:18.898824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40333 (local address 127.0.98.1:48763)
0927 00:45:18.899750 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:18.899790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:18.901195 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:18.901731 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:18.901737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:18.901818 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:18.902265 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.902277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.903068 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.903074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.904578 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:18.904591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:18.905178 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:18.905185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:18.905425 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:18.906072 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:18.906076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:18.906136 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:18.906221 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:18.906507 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:18.906519 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:18.906527 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:45:18.906679 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 00:45:18.906991 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":135,"threads_started":1}
I20260927 00:45:18.909369 18176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40333
I20260927 00:45:18.909631 18176 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:18.910172 18176 heartbeater.cc:511] Master 127.0.98.62:40333 requested a full tablet report, sending...
I20260927 00:45:18.911569 17988 ts_manager.cc:194] Registered new tserver with Master: 8ecd347926d1456faf2adb81cad2b0ba (127.0.98.1:41597)
I20260927 00:45:18.912667 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48763
W20260927 00:45:19.009621 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:19.009873 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:19.009948 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:19.013909 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:19.013976 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:19.014074 18179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:19.018750 18179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:19.019738 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:19.021070 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:19.030175 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:19.030207 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:19.030189 18179 server_base.cc:1034] running on GCE node
W20260927 00:45:19.030182 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:19.030753 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:19.031309 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:19.032522 18179 hybrid_clock.cc:648] HybridClock initialized: now 1790469919032512 us; error 28 us; skew 500 ppm
I20260927 00:45:19.034179 18179 webserver.cc:460] Webserver started at http://127.0.98.2:41867/ using document root <none> and password file <none>
I20260927 00:45:19.034727 18179 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:19.034814 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:19.035030 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:19.037097 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "240d680a8cb641cdbdf08b25795f15e0"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "28ef7d63b1d2377184ca1b92ccc74c3d"
server_key_iv: "7a52b101baf882794f1f0fee68cd597e"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.037683 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "240d680a8cb641cdbdf08b25795f15e0"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "28ef7d63b1d2377184ca1b92ccc74c3d"
server_key_iv: "7a52b101baf882794f1f0fee68cd597e"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.041325 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:19.043555 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.044694 18179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:19.044836 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "240d680a8cb641cdbdf08b25795f15e0"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "28ef7d63b1d2377184ca1b92ccc74c3d"
server_key_iv: "7a52b101baf882794f1f0fee68cd597e"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.044996 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:19.074618 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:19.075342 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:19.075546 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:19.076130 18179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:19.077134 18179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:19.077204 18179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.077275 18179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:19.077323 18179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.086452 18179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45017
I20260927 00:45:19.086470 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45017 every 8 connection(s)
I20260927 00:45:19.087476 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:19.096133 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.089324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50505 (local address 127.0.98.62:40333)
0927 00:45:19.090526 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.090530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.090548 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:45:19.091856 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.091858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.091900 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.091974 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:19.092747 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.093318 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.094385 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.094584 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.094665 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.095754 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:19.095756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:19.095758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:19.095768 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:19.095782 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:19.095788 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:19.095841 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.095974 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.096022 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120}
I20260927 00:45:19.096930 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.089757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40333 (local address 127.0.98.2:50505)
0927 00:45:19.090388 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.090438 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.091669 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.092097 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.092103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.092181 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:19.092633 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.092645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.093428 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.093430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.094283 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.094292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.094717 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.094720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.094853 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.095534 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:19.095535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:19.095574 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:19.095632 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:19.095853 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:19.095858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:19.095862 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.095960 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.096155 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260927 00:45:19.097301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18179
I20260927 00:45:19.097431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:19.097774 392 external_mini_cluster.cc:1428] Setting key 02c557499bf81d5baee031b8e6ed6617
I20260927 00:45:19.098071 18307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40333
I20260927 00:45:19.098299 18307 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:19.098858 18307 heartbeater.cc:511] Master 127.0.98.62:40333 requested a full tablet report, sending...
I20260927 00:45:19.099850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:19.099933 17988 ts_manager.cc:194] Registered new tserver with Master: 240d680a8cb641cdbdf08b25795f15e0 (127.0.98.2:45017)
I20260927 00:45:19.100481 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50505
W20260927 00:45:19.207559 18310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:19.207803 18310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:19.207840 18310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:19.212004 18310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:19.212056 18310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:19.212144 18310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:19.217038 18310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40333
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:19.218091 18310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:19.219436 18310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:19.228813 18316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:19.228771 18315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:19.228778 18318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:19.229012 18310 server_base.cc:1034] running on GCE node
I20260927 00:45:19.229507 18310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:19.230093 18310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:19.231287 18310 hybrid_clock.cc:648] HybridClock initialized: now 1790469919231269 us; error 29 us; skew 500 ppm
I20260927 00:45:19.233127 18310 webserver.cc:460] Webserver started at http://127.0.98.3:36059/ using document root <none> and password file <none>
I20260927 00:45:19.233673 18310 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:19.233767 18310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:19.233990 18310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:19.236029 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "b89b2a0098514d73bf03284cbdefbd81"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "94920c5aaba4445c90cb88f2f4d8d6b8"
server_key_iv: "3b5909d3a2241dd06f21c3da129aeff1"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.236598 18310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b89b2a0098514d73bf03284cbdefbd81"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "94920c5aaba4445c90cb88f2f4d8d6b8"
server_key_iv: "3b5909d3a2241dd06f21c3da129aeff1"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.240135 18310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:45:19.242532 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.243708 18310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:19.243841 18310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "b89b2a0098514d73bf03284cbdefbd81"
format_stamp: "Formatted at 2026-09-27 00:45:19 on dist-test-slave-pkws"
server_key: "94920c5aaba4445c90cb88f2f4d8d6b8"
server_key_iv: "3b5909d3a2241dd06f21c3da129aeff1"
server_key_version: "encryptionkey@0"
I20260927 00:45:19.243948 18310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:19.261258 18310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:19.261863 18310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:19.262049 18310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:19.262594 18310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:19.263476 18310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:19.263556 18310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.263644 18310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:19.263696 18310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:19.272333 18310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32833
I20260927 00:45:19.272349 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32833 every 8 connection(s)
I20260927 00:45:19.273216 18310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:19.275902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18310
I20260927 00:45:19.276006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:19.276328 392 external_mini_cluster.cc:1428] Setting key beb82670818e6e76bae1a2d8def2fc92
I20260927 00:45:19.282536 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.274966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47193 (local address 127.0.98.62:40333)
0927 00:45:19.275101 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.275104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.276222 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 00:45:19.277993 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.277996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.278047 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.278148 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:19.279081 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.279610 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.280717 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.280956 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.281066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.282058 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:19.282061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:19.282064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:19.282083 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:19.282100 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:19.282108 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:19.282181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.282337 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.282391 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:45:19.283294 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.275320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40333 (local address 127.0.98.3:47193)
0927 00:45:19.276053 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.276108 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.277767 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.278316 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.278322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.278402 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:19.278940 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.278959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.279733 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.279737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.280596 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.280605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.281098 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.281102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.281273 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.281805 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:19.281807 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:19.281842 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:19.281901 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:19.282194 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:19.282203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:19.282208 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.282336 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.282590 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260927 00:45:19.284327 18438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40333
I20260927 00:45:19.284528 18438 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:19.285017 18438 heartbeater.cc:511] Master 127.0.98.62:40333 requested a full tablet report, sending...
I20260927 00:45:19.285929 17988 ts_manager.cc:194] Registered new tserver with Master: b89b2a0098514d73bf03284cbdefbd81 (127.0.98.3:32833)
I20260927 00:45:19.286605 17988 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47193
I20260927 00:45:19.289996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:19.296739 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.292258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.0.98.62:40333)
0927 00:45:19.292376 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.292382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.293233 (+ 851us) server_negotiation.cc:407] Connection header received
0927 00:45:19.293351 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.293354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.293400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.293469 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:19.293837 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.294333 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.295175 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.295397 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.295505 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.296312 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:19.296316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:19.296318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:19.296330 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:19.296350 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:19.296365 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:19.296453 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.296546 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.296594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:19.299328 17988 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:19.302403 17988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:19.317018 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.310900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41597 (local address 127.0.0.1:59540)
0927 00:45:19.312222 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.312238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.312320 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.312811 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.312814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.312826 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:19.313019 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.313030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314157 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.314160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.315451 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.315461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.316425 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.316429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.316539 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.316578 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.316698 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.316812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":59,"thread_start_us":76,"threads_started":1}
I20260927 00:45:19.317075 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.311057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.0.98.1:41597)
0927 00:45:19.312313 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.312328 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.312358 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:45:19.312441 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.312446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.312630 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.312959 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:19.313105 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314022 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.315582 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.316288 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.316421 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.316629 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.316744 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.316844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":187,"threads_started":1}
I20260927 00:45:19.317824 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.311326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32833 (local address 127.0.0.1:54410)
0927 00:45:19.311837 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.311852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.311973 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.312986 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.312989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.313003 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:19.313105 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.313123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314576 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.314580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.316030 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.316037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.317269 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.317273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.317391 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.317440 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.317511 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.317591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":87,"spinlock_wait_cycles":7552,"thread_start_us":98,"threads_started":1}
I20260927 00:45:19.318065 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.311808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.98.3:32833)
0927 00:45:19.312552 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.312565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.312585 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:45:19.312660 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.312668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.312821 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.313018 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:19.313243 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314432 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.316147 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.316850 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.317020 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.317220 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.317756 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.317870 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":108,"threads_started":1}
I20260927 00:45:19.318243 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.311042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45017 (local address 127.0.0.1:53138)
0927 00:45:19.311776 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.311829 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.311973 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.312736 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.312746 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.312777 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:19.313281 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.313287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314615 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.314618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.315895 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.315905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.317274 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.317277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.317449 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.317476 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.317567 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.317655 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":442,"spinlock_wait_cycles":4608,"thread_start_us":74,"threads_started":1}
I20260927 00:45:19.318656 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.311667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.98.2:45017)
0927 00:45:19.312217 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.312230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.312260 (+ 30us) server_negotiation.cc:407] Connection header received
0927 00:45:19.312338 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.312346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.312552 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.312716 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:19.313396 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.314505 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.316045 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.316949 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.317059 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.317483 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.318381 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.318498 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":103,"threads_started":1}
I20260927 00:45:19.320493 18110 tablet_service.cc:1467] Processing CreateTablet for tablet bf3457cb68f74fae8614ca307c1775f0 (DEFAULT_TABLE table=test-table [id=129ce09178834a23b1f639d01761b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:19.320609 18372 tablet_service.cc:1467] Processing CreateTablet for tablet bf3457cb68f74fae8614ca307c1775f0 (DEFAULT_TABLE table=test-table [id=129ce09178834a23b1f639d01761b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:19.321192 18241 tablet_service.cc:1467] Processing CreateTablet for tablet bf3457cb68f74fae8614ca307c1775f0 (DEFAULT_TABLE table=test-table [id=129ce09178834a23b1f639d01761b705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:19.321604 18110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3457cb68f74fae8614ca307c1775f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:19.321586 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3457cb68f74fae8614ca307c1775f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:19.322054 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf3457cb68f74fae8614ca307c1775f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:19.327813 18455 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Bootstrap starting.
I20260927 00:45:19.329159 18456 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Bootstrap starting.
I20260927 00:45:19.330843 18456 tablet_bootstrap.cc:654] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:19.330857 18455 tablet_bootstrap.cc:654] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:19.331558 18456 log.cc:824] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:19.331570 18455 log.cc:824] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:19.332233 18457 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: Bootstrap starting.
I20260927 00:45:19.333443 18455 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: No bootstrap required, opened a new log
I20260927 00:45:19.333655 18455 ts_tablet_manager.cc:1397] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:45:19.334128 18456 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: No bootstrap required, opened a new log
I20260927 00:45:19.334168 18457 tablet_bootstrap.cc:654] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:19.334313 18456 ts_tablet_manager.cc:1397] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:45:19.334961 18457 log.cc:824] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:19.337364 18457 tablet_bootstrap.cc:492] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: No bootstrap required, opened a new log
I20260927 00:45:19.337546 18457 ts_tablet_manager.cc:1397] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 00:45:19.337744 18455 raft_consensus.cc:348] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.338003 18455 raft_consensus.cc:374] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:19.338078 18455 raft_consensus.cc:729] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89b2a0098514d73bf03284cbdefbd81, State: Initialized, Role: FOLLOWER
I20260927 00:45:19.338454 18456 raft_consensus.cc:348] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.338560 18455 consensus_queue.cc:260] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.338680 18456 raft_consensus.cc:374] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:19.338763 18456 raft_consensus.cc:729] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240d680a8cb641cdbdf08b25795f15e0, State: Initialized, Role: FOLLOWER
I20260927 00:45:19.339188 18456 consensus_queue.cc:260] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.339293 18438 heartbeater.cc:503] Master 127.0.98.62:40333 was elected leader, sending a full tablet report...
I20260927 00:45:19.339649 18455 ts_tablet_manager.cc:1428] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:45:19.340147 18456 ts_tablet_manager.cc:1428] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:45:19.340204 18307 heartbeater.cc:503] Master 127.0.98.62:40333 was elected leader, sending a full tablet report...
I20260927 00:45:19.341764 18457 raft_consensus.cc:348] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.342057 18457 raft_consensus.cc:374] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:19.342131 18457 raft_consensus.cc:729] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ecd347926d1456faf2adb81cad2b0ba, State: Initialized, Role: FOLLOWER
I20260927 00:45:19.342748 18457 consensus_queue.cc:260] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.343391 18176 heartbeater.cc:503] Master 127.0.98.62:40333 was elected leader, sending a full tablet report...
I20260927 00:45:19.344426 18457 ts_tablet_manager.cc:1428] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:45:19.356029 18463 raft_consensus.cc:482] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:19.356215 18463 raft_consensus.cc:504] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.357239 18463 leader_election.cc:288] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 240d680a8cb641cdbdf08b25795f15e0 (127.0.98.2:45017), b89b2a0098514d73bf03284cbdefbd81 (127.0.98.3:32833)
I20260927 00:45:19.362071 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.357569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36977 (local address 127.0.98.2:45017)
0927 00:45:19.357701 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.357707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.357725 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:45:19.357869 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.357872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.357915 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.358012 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:19.358320 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.359103 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.360753 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.361356 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.361467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.361527 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.361862 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.361943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 00:45:19.362095 18178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.357465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45017 (local address 127.0.98.1:36977)
0927 00:45:19.357629 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.357648 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.357737 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.358044 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.358050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.358070 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:19.358181 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.358187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.359210 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.359212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.360609 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.360622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.361602 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.361606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.361713 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.361745 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.361823 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.361931 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":86}
I20260927 00:45:19.362110 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.357578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34809 (local address 127.0.98.3:32833)
0927 00:45:19.357708 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.357713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.357986 (+ 273us) server_negotiation.cc:407] Connection header received
0927 00:45:19.358120 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.358124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.358169 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.358227 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:19.358498 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.359211 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.360964 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.361562 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.361660 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.361715 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.361934 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.362010 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:45:19.362100 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.357469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32833 (local address 127.0.98.1:34809)
0927 00:45:19.357897 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.357907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:19.358017 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:19.358266 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:19.358272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:19.358292 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:19.358401 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.358405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.359310 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.359313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.360609 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:19.360621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.361683 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.361686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:19.361765 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.361785 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:45:19.361885 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 00:45:19.361950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":161,"thread_start_us":113,"threads_started":1}
I20260927 00:45:19.362752 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3457cb68f74fae8614ca307c1775f0" candidate_uuid: "8ecd347926d1456faf2adb81cad2b0ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240d680a8cb641cdbdf08b25795f15e0" is_pre_election: true
I20260927 00:45:19.362761 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3457cb68f74fae8614ca307c1775f0" candidate_uuid: "8ecd347926d1456faf2adb81cad2b0ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89b2a0098514d73bf03284cbdefbd81" is_pre_election: true
I20260927 00:45:19.363025 18392 raft_consensus.cc:2454] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ecd347926d1456faf2adb81cad2b0ba in term 0.
I20260927 00:45:19.363035 18261 raft_consensus.cc:2454] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ecd347926d1456faf2adb81cad2b0ba in term 0.
I20260927 00:45:19.363492 18063 leader_election.cc:302] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ecd347926d1456faf2adb81cad2b0ba, b89b2a0098514d73bf03284cbdefbd81; no voters:
I20260927 00:45:19.363763 18463 raft_consensus.cc:2781] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:19.363837 18463 raft_consensus.cc:482] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:19.363866 18463 raft_consensus.cc:3037] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:19.364823 18463 raft_consensus.cc:504] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.365252 18463 leader_election.cc:288] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [CANDIDATE]: Term 1 election: Requested vote from peers 240d680a8cb641cdbdf08b25795f15e0 (127.0.98.2:45017), b89b2a0098514d73bf03284cbdefbd81 (127.0.98.3:32833)
I20260927 00:45:19.365620 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3457cb68f74fae8614ca307c1775f0" candidate_uuid: "8ecd347926d1456faf2adb81cad2b0ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89b2a0098514d73bf03284cbdefbd81"
I20260927 00:45:19.365603 18261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf3457cb68f74fae8614ca307c1775f0" candidate_uuid: "8ecd347926d1456faf2adb81cad2b0ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "240d680a8cb641cdbdf08b25795f15e0"
I20260927 00:45:19.365751 18261 raft_consensus.cc:3037] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:19.365767 18392 raft_consensus.cc:3037] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:19.366777 18261 raft_consensus.cc:2454] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ecd347926d1456faf2adb81cad2b0ba in term 1.
I20260927 00:45:19.366791 18392 raft_consensus.cc:2454] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ecd347926d1456faf2adb81cad2b0ba in term 1.
I20260927 00:45:19.367092 18066 leader_election.cc:302] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240d680a8cb641cdbdf08b25795f15e0, 8ecd347926d1456faf2adb81cad2b0ba; no voters:
I20260927 00:45:19.367326 18463 raft_consensus.cc:2781] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:19.367501 18463 raft_consensus.cc:686] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [term 1 LEADER]: Becoming Leader. State: Replica: 8ecd347926d1456faf2adb81cad2b0ba, State: Running, Role: LEADER
I20260927 00:45:19.367806 18463 consensus_queue.cc:237] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } }
I20260927 00:45:19.370723 17987 catalog_manager.cc:5495] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ecd347926d1456faf2adb81cad2b0ba (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8ecd347926d1456faf2adb81cad2b0ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ecd347926d1456faf2adb81cad2b0ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 } health_report { overall_health: UNKNOWN } } }
I20260927 00:45:19.381639 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.378172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.0.98.1:41597)
0927 00:45:19.378282 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:19.378286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:19.378297 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:45:19.378410 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:19.378432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:19.378512 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:19.378599 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:19.378830 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.379331 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.380254 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:19.380563 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:19.380639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:19.380936 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:19.381325 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:19.381419 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:19.381486 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 00:45:19.381527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:45:19.387548 18392 raft_consensus.cc:1261] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 1 FOLLOWER]: Refusing update from remote peer 8ecd347926d1456faf2adb81cad2b0ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:19.387548 18261 raft_consensus.cc:1261] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 1 FOLLOWER]: Refusing update from remote peer 8ecd347926d1456faf2adb81cad2b0ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:19.388345 18465 consensus_queue.cc:1035] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89b2a0098514d73bf03284cbdefbd81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:19.388568 18463 consensus_queue.cc:1035] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "240d680a8cb641cdbdf08b25795f15e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:19.397565 18470 mvcc.cc:205] Tried to move back new op lower bound from 7333764789804023808 to 7333764789733040128. Current Snapshot: MvccSnapshot[applied={T|T < 7333764789804023808}]
I20260927 00:45:19.397771 18469 mvcc.cc:205] Tried to move back new op lower bound from 7333764789804023808 to 7333764789733040128. Current Snapshot: MvccSnapshot[applied={T|T < 7333764789804023808 or (T in {7333764789804023808})}]
I20260927 00:45:19.405086 17988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:19.405258 17988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:19.407727 17988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26fbf3bb28ad433faf3faa16bf66c75c: Sending DeleteTablet for 3 replicas of tablet bf3457cb68f74fae8614ca307c1775f0
I20260927 00:45:19.408532 18241 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3457cb68f74fae8614ca307c1775f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:19 UTC) from {username='slave'} at 127.0.0.1:53138
I20260927 00:45:19.408656 18372 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3457cb68f74fae8614ca307c1775f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:19 UTC) from {username='slave'} at 127.0.0.1:54410
I20260927 00:45:19.408658 18110 tablet_service.cc:1514] Processing DeleteTablet for tablet bf3457cb68f74fae8614ca307c1775f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:19 UTC) from {username='slave'} at 127.0.0.1:59540
I20260927 00:45:19.409166 18478 tablet_replica.cc:330] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: stopping tablet replica
I20260927 00:45:19.409451 18480 tablet_replica.cc:330] T bf3457cb68f74fae8614ca307c1775f0 P 8ecd347926d1456faf2adb81cad2b0ba: stopping tablet replica
I20260927 00:45:19.409667 18478 raft_consensus.cc:2229] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:19.410080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18048
I20260927 00:45:19.411469 18479 tablet_replica.cc:330] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: stopping tablet replica
I20260927 00:45:19.411762 18479 raft_consensus.cc:2229] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:19.412046 18479 raft_consensus.cc:2258] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:19.412070 18478 raft_consensus.cc:2258] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:19.413489 18479 ts_tablet_manager.cc:1905] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:19.413553 18478 ts_tablet_manager.cc:1905] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:19.415639 18479 ts_tablet_manager.cc:1918] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:19.415750 18479 log.cc:1196] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/bf3457cb68f74fae8614ca307c1775f0
I20260927 00:45:19.416025 18479 ts_tablet_manager.cc:1939] T bf3457cb68f74fae8614ca307c1775f0 P b89b2a0098514d73bf03284cbdefbd81: Deleting consensus metadata
I20260927 00:45:19.416915 18478 ts_tablet_manager.cc:1918] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:19.417016 18478 log.cc:1196] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/bf3457cb68f74fae8614ca307c1775f0
I20260927 00:45:19.417200 17972 catalog_manager.cc:4841] TS b89b2a0098514d73bf03284cbdefbd81 (127.0.98.3:32833): tablet bf3457cb68f74fae8614ca307c1775f0 (table test-table [id=129ce09178834a23b1f639d01761b705]) successfully deleted
I20260927 00:45:19.417308 18478 ts_tablet_manager.cc:1939] T bf3457cb68f74fae8614ca307c1775f0 P 240d680a8cb641cdbdf08b25795f15e0: Deleting consensus metadata
W20260927 00:45:19.417614 17973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41597 (error 108)
I20260927 00:45:19.417860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18179
I20260927 00:45:19.420058 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.419708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41597 (local address 127.0.0.1:59556)
0927 00:45:19.419866 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.419951 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:45:19.420226 17973 catalog_manager.cc:4584] TS 8ecd347926d1456faf2adb81cad2b0ba (127.0.98.1:41597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf3457cb68f74fae8614ca307c1775f0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41597: connect: Connection refused (error 111)
I20260927 00:45:19.423636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18310
I20260927 00:45:19.424729 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:19.423691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45017 (local address 127.0.0.1:53154)
0927 00:45:19.424564 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:19.424631 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":769}
I20260927 00:45:19.429370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 17957
2026-09-27T00:45:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:19 dist-test-slave-pkws krb5kdc[18484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:19 dist-test-slave-pkws krb5kdc[18484](info): set up 2 sockets
Sep 27 00:45:19 dist-test-slave-pkws krb5kdc[18484](info): commencing operation
krb5kdc: starting...
W20260927 00:45:20.480688 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:20 dist-test-slave-pkws krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:20.600274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:20.706369 18497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:20.706642 18497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:20.706727 18497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:20.710650 18497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:20.710736 18497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:20.710774 18497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:20.710834 18497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:20.710872 18497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:20.715958 18497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:20.716964 18497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:20.717941 18497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:20.725769 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:20.725852 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:20.725792 18502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:20.726022 18497 server_base.cc:1034] running on GCE node
I20260927 00:45:20.726677 18497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:20.727671 18497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:20.728832 18497 hybrid_clock.cc:648] HybridClock initialized: now 1790469920728813 us; error 29 us; skew 500 ppm
Sep 27 00:45:20 dist-test-slave-pkws krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:20.731992 18497 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:20.732751 18497 webserver.cc:460] Webserver started at http://127.0.98.62:39209/ using document root <none> and password file <none>
I20260927 00:45:20.733304 18497 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:20.733359 18497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:20.733517 18497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:20.735472 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "5fec66d96fd0427281c17b7d13689b23"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "711f7b7ad3f9cff7df1c6ba253b19479"
server_key_iv: "42498c8d60cae1a18894954a62877658"
server_key_version: "encryptionkey@0"
I20260927 00:45:20.735960 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "5fec66d96fd0427281c17b7d13689b23"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "711f7b7ad3f9cff7df1c6ba253b19479"
server_key_iv: "42498c8d60cae1a18894954a62877658"
server_key_version: "encryptionkey@0"
I20260927 00:45:20.739360 18497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:20.741600 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:20.742743 18497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:20.742885 18497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "5fec66d96fd0427281c17b7d13689b23"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "711f7b7ad3f9cff7df1c6ba253b19479"
server_key_iv: "42498c8d60cae1a18894954a62877658"
server_key_version: "encryptionkey@0"
I20260927 00:45:20.742993 18497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:20.757246 18497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:20.759822 18497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:20.760012 18497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:20.766968 18497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39049
I20260927 00:45:20.767038 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39049 every 8 connection(s)
I20260927 00:45:20.767874 18497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:20.770495 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:20.775552 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: Bootstrap starting.
I20260927 00:45:20.775835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18497
I20260927 00:45:20.775990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:20.776257 392 external_mini_cluster.cc:1428] Setting key 5b355150f9d3e5ddf5364188799bbe53
I20260927 00:45:20.778019 18565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:20.778717 18565 log.cc:824] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:20.780436 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: No bootstrap required, opened a new log
I20260927 00:45:20.783891 18565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } }
I20260927 00:45:20.784081 18565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:20.784166 18565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fec66d96fd0427281c17b7d13689b23, State: Initialized, Role: FOLLOWER
Sep 27 00:45:20 dist-test-slave-pkws krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:20.784559 18565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } }
I20260927 00:45:20.784691 18565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:20.784781 18565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:20.784915 18565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:20.785846 18565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } }
I20260927 00:45:20.786242 18565 leader_election.cc:302] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fec66d96fd0427281c17b7d13689b23; no voters:
I20260927 00:45:20.786542 18565 leader_election.cc:288] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:20.786648 18570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:20.786851 18570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [term 1 LEADER]: Becoming Leader. State: Replica: 5fec66d96fd0427281c17b7d13689b23, State: Running, Role: LEADER
I20260927 00:45:20.787151 18570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } }
I20260927 00:45:20.787832 18565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:20.790735 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fec66d96fd0427281c17b7d13689b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } } }
I20260927 00:45:20.790800 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:20.777489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.0.98.62:39049)
0927 00:45:20.777849 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:20.777869 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:20.777903 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:45:20.778534 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:20.778551 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:20.778900 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:20.779215 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:20.780043 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:20.781077 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:20.781926 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:20.782397 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:20.782565 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:20.785234 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:20.785240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:20.785254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:20.785286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:20.787289 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:20.787773 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:20.787778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:20.787784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:20.787914 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:20.788248 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:20.788251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:20.788252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:20.788677 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:20.788862 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:20.789140 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 00:45:20.789437 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20260927 00:45:20.790875 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:20.791357 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fec66d96fd0427281c17b7d13689b23. Latest consensus state: current_term: 1 leader_uuid: "5fec66d96fd0427281c17b7d13689b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fec66d96fd0427281c17b7d13689b23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39049 } } }
I20260927 00:45:20.791450 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23 [sys.catalog]: This master's current role is: LEADER
W20260927 00:45:20.793753 18585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:20.793817 18585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:20.793897 18586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:20.794531 18586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:20.799306 18586 catalog_manager.cc:1347] Generated new cluster ID: c1b4cfb41cde4af1a6dd313dae8653f9
I20260927 00:45:20.799365 18586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:20.817346 18586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:20.818506 18586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:20.831650 18586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: Generated new TSK 0
I20260927 00:45:20.832389 18586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:45:20.863507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:20.970685 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:20.970965 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:20.971040 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:20.975044 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:20.975126 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:20.975217 18591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:20.980278 18591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:20.981324 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:20.982303 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:20.991802 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:20.991817 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:20.991953 18591 server_base.cc:1034] running on GCE node
W20260927 00:45:20.991796 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:20.992411 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:20.993376 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:20.994552 18591 hybrid_clock.cc:648] HybridClock initialized: now 1790469920994537 us; error 30 us; skew 500 ppm
Sep 27 00:45:20 dist-test-slave-pkws krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:20.997279 18591 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:45:20.998087 18591 webserver.cc:460] Webserver started at http://127.0.98.1:41927/ using document root <none> and password file <none>
I20260927 00:45:20.998687 18591 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:20.998780 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:20.998986 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:21.001006 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9b9d3dc329945d69d75197d1c37dce2"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "877c6e0e872b79fa4f3ffe2fccd0e379"
server_key_iv: "8a465d33e54bad5bba0fbb3b11f5a936"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.001573 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9b9d3dc329945d69d75197d1c37dce2"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "877c6e0e872b79fa4f3ffe2fccd0e379"
server_key_iv: "8a465d33e54bad5bba0fbb3b11f5a936"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.005360 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 00:45:21.007755 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.009084 18591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:21.009209 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "b9b9d3dc329945d69d75197d1c37dce2"
format_stamp: "Formatted at 2026-09-27 00:45:20 on dist-test-slave-pkws"
server_key: "877c6e0e872b79fa4f3ffe2fccd0e379"
server_key_iv: "8a465d33e54bad5bba0fbb3b11f5a936"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.009315 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:21.023149 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:21.025952 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:21.026153 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:21.026695 18591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:21.027473 18591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:21.027544 18591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.027611 18591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:21.027666 18591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.036670 18591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39315
I20260927 00:45:21.036697 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39315 every 8 connection(s)
I20260927 00:45:21.037562 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:21.039443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18591
I20260927 00:45:21.039595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:21.039930 392 external_mini_cluster.cc:1428] Setting key ad564424ad0153d06515d405e6fac953
Sep 27 00:45:21 dist-test-slave-pkws krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 00:45:21.051860 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.039290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45761 (local address 127.0.98.62:39049)
0927 00:45:21.039457 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.039465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.040474 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 00:45:21.041668 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.041671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.041721 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.041800 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:21.042973 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.043508 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.044661 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.044970 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.045078 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.047858 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:21.047862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:21.047865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:21.047898 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:21.049451 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.050114 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.050119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.050121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.050203 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.050585 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.050589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.050591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.050855 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:21.050991 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.051569 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.051708 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:45:21.052682 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.039759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39049 (local address 127.0.98.1:45761)
0927 00:45:21.040282 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.040348 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.041373 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.041970 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.041977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.042317 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:21.042809 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.042821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.043635 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.043638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.044523 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.044532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.045110 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.045114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.045281 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.045887 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:21.045902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:21.047700 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:21.049601 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.049607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.049621 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.050012 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.050303 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.050306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.050308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.050451 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.051013 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:21.051019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:21.051246 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.051489 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.051852 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260927 00:45:21.054145 18720 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39049
I20260927 00:45:21.054392 18720 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:21.054960 18720 heartbeater.cc:511] Master 127.0.98.62:39049 requested a full tablet report, sending...
I20260927 00:45:21.056933 18527 ts_manager.cc:194] Registered new tserver with Master: b9b9d3dc329945d69d75197d1c37dce2 (127.0.98.1:39315)
I20260927 00:45:21.058611 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:45:21.068289 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:21.175308 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:21.175547 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:21.175587 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:21.179607 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:21.179659 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:21.179770 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:21.184794 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:21.185753 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:21.186697 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:21.196537 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:21.196568 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:21.196620 18725 server_base.cc:1034] running on GCE node
W20260927 00:45:21.196550 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:21.197181 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:21.197801 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:21.198988 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790469921198975 us; error 24 us; skew 500 ppm
Sep 27 00:45:21 dist-test-slave-pkws krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:21.201654 18725 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:45:21.202488 18725 webserver.cc:460] Webserver started at http://127.0.98.2:39861/ using document root <none> and password file <none>
I20260927 00:45:21.203018 18725 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:21.203104 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:21.203320 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:21.205253 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "177f5dd5ce12ae450ff7397e9f3c9f42"
server_key_iv: "3624aa5307a75d6e31fbeacf732eb1d9"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.205806 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "177f5dd5ce12ae450ff7397e9f3c9f42"
server_key_iv: "3624aa5307a75d6e31fbeacf732eb1d9"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.209298 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:21.211557 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.212792 18725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:21.212956 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "177f5dd5ce12ae450ff7397e9f3c9f42"
server_key_iv: "3624aa5307a75d6e31fbeacf732eb1d9"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.213078 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:21.223587 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:21.226212 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:21.226398 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:21.226920 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:21.227670 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:21.227741 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.227807 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:21.227861 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.236541 18725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38369
I20260927 00:45:21.236572 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38369 every 8 connection(s)
I20260927 00:45:21.237386 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:21.243598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18725
I20260927 00:45:21.243738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:21.244032 392 external_mini_cluster.cc:1428] Setting key 3d5577ffe438846f25dd1354b516b568
Sep 27 00:45:21 dist-test-slave-pkws krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:21.250761 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.239004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33983 (local address 127.0.98.62:39049)
0927 00:45:21.239134 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.239138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.239849 (+ 711us) server_negotiation.cc:407] Connection header received
0927 00:45:21.240661 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.240663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.240712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.240785 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:21.241978 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.242523 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.243839 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.244090 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.244221 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.246855 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:21.246862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:21.246868 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:21.246902 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:21.248538 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.249126 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.249131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.249133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.249232 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.249599 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.249603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.249605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.249823 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:21.249943 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.250445 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.250572 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:45:21.251498 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.239280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39049 (local address 127.0.98.2:33983)
0927 00:45:21.239692 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.239729 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.240469 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.240998 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.241005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.241405 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:21.241832 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.241844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.242638 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.242641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.243706 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.243719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.244254 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.244258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.244463 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.245116 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:21.245131 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:21.246684 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:21.248647 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.248652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.248662 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.249027 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.249336 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.249339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.249344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.249481 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.249952 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:21.249960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:21.250190 (+ 230us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.250373 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.250693 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260927 00:45:21.252717 18854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39049
I20260927 00:45:21.253008 18854 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:21.253464 18854 heartbeater.cc:511] Master 127.0.98.62:39049 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:45:21.254534 18527 ts_manager.cc:194] Registered new tserver with Master: 74d4dcb49bdd4a2e8f0a8c7c402547db (127.0.98.2:38369)
I20260927 00:45:21.255115 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:45:21.271864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:21.381120 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:21.381343 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:21.381384 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:21.385291 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:21.385341 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:21.385394 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:21.390789 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:21.391768 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:21.392722 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:21.402345 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:21.402455 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:21.402352 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:21.402710 18859 server_base.cc:1034] running on GCE node
I20260927 00:45:21.403090 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:21.403846 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:21.404954 18859 hybrid_clock.cc:648] HybridClock initialized: now 1790469921404929 us; error 39 us; skew 500 ppm
Sep 27 00:45:21 dist-test-slave-pkws krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:21.415604 18859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:45:21.416590 18859 webserver.cc:460] Webserver started at http://127.0.98.3:32899/ using document root <none> and password file <none>
I20260927 00:45:21.417273 18859 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:21.417366 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:21.417579 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:21.419579 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "7bbb399e30fd43509d2e7d355ededfc6"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "3026d156b313cf0772b96a477dc01149"
server_key_iv: "e14eb54a662efce5259fbf166559b7ce"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.420140 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7bbb399e30fd43509d2e7d355ededfc6"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "3026d156b313cf0772b96a477dc01149"
server_key_iv: "e14eb54a662efce5259fbf166559b7ce"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.423744 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 00:45:21.426137 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.427426 18859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:21.427572 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "7bbb399e30fd43509d2e7d355ededfc6"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "3026d156b313cf0772b96a477dc01149"
server_key_iv: "e14eb54a662efce5259fbf166559b7ce"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.427681 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:21.444013 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:21.446959 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:21.447139 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:21.447676 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:21.448447 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:21.448524 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.448593 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:21.448629 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.457677 18859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36285
I20260927 00:45:21.457728 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36285 every 8 connection(s)
I20260927 00:45:21.458539 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:21.467708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 18859
Sep 27 00:45:21 dist-test-slave-pkws krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:21.467813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:21.468143 392 external_mini_cluster.cc:1428] Setting key 1a0cfb7c9939e52d5893406d57ea3b63
I20260927 00:45:21.472324 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.460345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45697 (local address 127.0.98.62:39049)
0927 00:45:21.460541 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.460546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.461408 (+ 862us) server_negotiation.cc:407] Connection header received
0927 00:45:21.462333 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.462336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.462390 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.462484 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:21.463677 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.464256 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.465579 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.465812 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.465910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.468475 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:21.468479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:21.468481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:21.468513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:21.470450 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.470989 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.470992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.470993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.471071 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.471390 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.471394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.471396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.471583 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:21.471673 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.472115 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.472207 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 00:45:21.473111 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.460736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39049 (local address 127.0.98.3:45697)
0927 00:45:21.461246 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.461288 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.462108 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.462643 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.462649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.463031 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:21.463515 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.463529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.464413 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.464415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.465447 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.465457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.465950 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.465952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.466094 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.466725 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:21.466741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:21.468299 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:21.470592 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.470597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.470611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.470895 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.471153 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:21.471155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:21.471157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:21.471279 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:21.471682 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:21.471686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:21.471910 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.472059 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.472284 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":173,"threads_started":1}
I20260927 00:45:21.474148 18988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39049
I20260927 00:45:21.474380 18988 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:21.474834 18988 heartbeater.cc:511] Master 127.0.98.62:39049 requested a full tablet report, sending...
I20260927 00:45:21.475795 18527 ts_manager.cc:194] Registered new tserver with Master: 7bbb399e30fd43509d2e7d355ededfc6 (127.0.98.3:36285)
I20260927 00:45:21.476372 18527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45697
I20260927 00:45:21.482251 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:21.492795 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.484143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.0.98.62:39049)
0927 00:45:21.484291 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.484295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.484468 (+ 173us) server_negotiation.cc:407] Connection header received
0927 00:45:21.484620 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.484623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.484672 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.484756 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:21.485555 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.486100 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.486994 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.487239 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.487355 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.489188 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:21.489192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:21.489195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:21.489221 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:21.490875 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.491417 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.491420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.491422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.491510 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:21.491860 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:21.491864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:21.491866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:21.492076 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:21.492242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.492526 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.492621 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:21.495765 18527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:21.499017 18527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:21.513141 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.507013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36285 (local address 127.0.0.1:47180)
0927 00:45:21.507341 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.507375 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.507521 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.508246 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.508254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.508280 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:21.508406 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.508414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.509768 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.509775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.511292 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.511303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.512525 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.512528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.512646 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.512687 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.512828 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.512991 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
I20260927 00:45:21.513599 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.508004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.1:39315)
0927 00:45:21.508396 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.508408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.508714 (+ 306us) server_negotiation.cc:407] Connection header received
0927 00:45:21.508831 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.508835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.509028 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.509428 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:21.509552 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.510344 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.511833 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.512795 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.512938 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.513084 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.513140 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.513448 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":105,"threads_started":1}
I20260927 00:45:21.513600 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.507212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.0.98.3:36285)
0927 00:45:21.507587 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.507599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.507621 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:45:21.507691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.507696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.507895 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.508056 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:21.508509 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.509660 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.511446 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.512201 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.512313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.512455 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.513334 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.513438 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":158,"threads_started":1}
I20260927 00:45:21.513635 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.507855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39315 (local address 127.0.0.1:42356)
0927 00:45:21.508595 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.508609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.508716 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.509198 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.509201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.509219 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:21.509300 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.509305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.510487 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.510493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.511699 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.511709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.512918 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.512922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.513023 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.513049 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.513116 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.513182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260927 00:45:21.514425 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.508606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.0.98.2:38369)
0927 00:45:21.509100 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.509115 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.509142 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:45:21.509226 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.509230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.509367 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.509524 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:21.509812 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.511256 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.512434 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.513269 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.513358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.513473 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.514151 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.514242 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":94,"threads_started":1}
I20260927 00:45:21.514711 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.508177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38369 (local address 127.0.0.1:54862)
0927 00:45:21.508979 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.508992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.509093 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.509510 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.509513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.509532 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:21.509681 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.509688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.511398 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.511401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.512299 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.512310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.513530 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.513534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.513630 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.513660 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.513731 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.514629 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":88,"threads_started":1}
I20260927 00:45:21.516717 18654 tablet_service.cc:1467] Processing CreateTablet for tablet 43bac2896d814a50b4ce41f8edaad01d (DEFAULT_TABLE table=test-table [id=6dce48de6cc947b3bd35b71b08dded82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:21.516695 18922 tablet_service.cc:1467] Processing CreateTablet for tablet 43bac2896d814a50b4ce41f8edaad01d (DEFAULT_TABLE table=test-table [id=6dce48de6cc947b3bd35b71b08dded82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:21.517211 18788 tablet_service.cc:1467] Processing CreateTablet for tablet 43bac2896d814a50b4ce41f8edaad01d (DEFAULT_TABLE table=test-table [id=6dce48de6cc947b3bd35b71b08dded82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:21.517827 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bac2896d814a50b4ce41f8edaad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:21.517834 18654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bac2896d814a50b4ce41f8edaad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:21.518034 18788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43bac2896d814a50b4ce41f8edaad01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:21.523738 19005 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Bootstrap starting.
I20260927 00:45:21.524621 19006 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Bootstrap starting.
I20260927 00:45:21.525183 19007 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Bootstrap starting.
I20260927 00:45:21.527086 19005 tablet_bootstrap.cc:654] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:21.527174 19007 tablet_bootstrap.cc:654] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:21.527590 19006 tablet_bootstrap.cc:654] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:21.528200 19007 log.cc:824] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:21.528411 19006 log.cc:824] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:21.528411 19005 log.cc:824] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:21.530061 19007 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: No bootstrap required, opened a new log
I20260927 00:45:21.530277 19007 ts_tablet_manager.cc:1397] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:45:21.530301 19006 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: No bootstrap required, opened a new log
I20260927 00:45:21.530303 19005 tablet_bootstrap.cc:492] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: No bootstrap required, opened a new log
I20260927 00:45:21.530474 19005 ts_tablet_manager.cc:1397] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 00:45:21.530468 19006 ts_tablet_manager.cc:1397] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:45:21.533885 19007 raft_consensus.cc:348] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.534123 19007 raft_consensus.cc:374] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:21.534201 19007 raft_consensus.cc:729] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b9d3dc329945d69d75197d1c37dce2, State: Initialized, Role: FOLLOWER
I20260927 00:45:21.534279 19006 raft_consensus.cc:348] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.534279 19005 raft_consensus.cc:348] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.534550 19006 raft_consensus.cc:374] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:21.534608 19006 raft_consensus.cc:729] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d4dcb49bdd4a2e8f0a8c7c402547db, State: Initialized, Role: FOLLOWER
I20260927 00:45:21.534633 19005 raft_consensus.cc:374] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:21.534704 19005 raft_consensus.cc:729] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bbb399e30fd43509d2e7d355ededfc6, State: Initialized, Role: FOLLOWER
I20260927 00:45:21.534721 19007 consensus_queue.cc:260] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.535053 19005 consensus_queue.cc:260] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.535053 19006 consensus_queue.cc:260] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.535418 18720 heartbeater.cc:503] Master 127.0.98.62:39049 was elected leader, sending a full tablet report...
I20260927 00:45:21.536022 19006 ts_tablet_manager.cc:1428] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:45:21.536022 19005 ts_tablet_manager.cc:1428] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:45:21.536370 18988 heartbeater.cc:503] Master 127.0.98.62:39049 was elected leader, sending a full tablet report...
I20260927 00:45:21.536442 18854 heartbeater.cc:503] Master 127.0.98.62:39049 was elected leader, sending a full tablet report...
I20260927 00:45:21.537632 19007 ts_tablet_manager.cc:1428] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 00:45:21.538618 18721 tablet.cc:2369] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:21.661918 19013 raft_consensus.cc:482] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:21.662149 19013 raft_consensus.cc:504] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.663272 19013 leader_election.cc:288] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9b9d3dc329945d69d75197d1c37dce2 (127.0.98.1:39315), 74d4dcb49bdd4a2e8f0a8c7c402547db (127.0.98.2:38369)
I20260927 00:45:21.667287 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.663554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39315 (local address 127.0.98.3:54825)
0927 00:45:21.663698 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.663714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.663813 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.664203 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.664205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.664219 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:21.664306 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.664310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.665225 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.665228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.666144 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.666152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.666880 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.666884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.666991 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.667018 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.667096 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.667163 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 00:45:21.667323 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.663685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54825 (local address 127.0.98.1:39315)
0927 00:45:21.663847 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.663852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.663870 (+ 18us) server_negotiation.cc:407] Connection header received
0927 00:45:21.663934 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.663940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.664000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.664080 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:21.664398 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.665116 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.666268 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.666762 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.666841 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.666889 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.667128 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.667196 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 00:45:21.667552 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.663837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38369 (local address 127.0.98.3:55765)
0927 00:45:21.664203 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.664213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:21.664326 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:21.664617 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:21.664619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:21.664637 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:21.664754 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.664759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.665562 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.665565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.666417 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:21.666426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.667139 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.667143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:21.667236 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.667258 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:45:21.667336 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 00:45:21.667406 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":130,"threads_started":1}
I20260927 00:45:21.667593 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.663980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55765 (local address 127.0.98.2:38369)
0927 00:45:21.664113 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.664119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.664288 (+ 169us) server_negotiation.cc:407] Connection header received
0927 00:45:21.664459 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.664463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.664513 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.664570 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:21.664858 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.665452 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.666539 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.667013 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.667118 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.667171 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.667372 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.667449 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 00:45:21.668087 18674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bac2896d814a50b4ce41f8edaad01d" candidate_uuid: "7bbb399e30fd43509d2e7d355ededfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b9d3dc329945d69d75197d1c37dce2" is_pre_election: true
I20260927 00:45:21.668370 18674 raft_consensus.cc:2454] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bbb399e30fd43509d2e7d355ededfc6 in term 0.
I20260927 00:45:21.668371 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bac2896d814a50b4ce41f8edaad01d" candidate_uuid: "7bbb399e30fd43509d2e7d355ededfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" is_pre_election: true
I20260927 00:45:21.668702 18808 raft_consensus.cc:2454] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bbb399e30fd43509d2e7d355ededfc6 in term 0.
I20260927 00:45:21.668942 18876 leader_election.cc:302] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bbb399e30fd43509d2e7d355ededfc6, b9b9d3dc329945d69d75197d1c37dce2; no voters:
I20260927 00:45:21.669229 19013 raft_consensus.cc:2781] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:21.669317 19013 raft_consensus.cc:482] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:21.669361 19013 raft_consensus.cc:3037] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:21.670380 19013 raft_consensus.cc:504] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.670711 19013 leader_election.cc:288] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [CANDIDATE]: Term 1 election: Requested vote from peers b9b9d3dc329945d69d75197d1c37dce2 (127.0.98.1:39315), 74d4dcb49bdd4a2e8f0a8c7c402547db (127.0.98.2:38369)
I20260927 00:45:21.671031 18674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bac2896d814a50b4ce41f8edaad01d" candidate_uuid: "7bbb399e30fd43509d2e7d355ededfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b9d3dc329945d69d75197d1c37dce2"
I20260927 00:45:21.671077 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43bac2896d814a50b4ce41f8edaad01d" candidate_uuid: "7bbb399e30fd43509d2e7d355ededfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db"
I20260927 00:45:21.671132 18674 raft_consensus.cc:3037] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:21.671211 18808 raft_consensus.cc:3037] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:21.672147 18674 raft_consensus.cc:2454] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bbb399e30fd43509d2e7d355ededfc6 in term 1.
I20260927 00:45:21.672293 18808 raft_consensus.cc:2454] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bbb399e30fd43509d2e7d355ededfc6 in term 1.
I20260927 00:45:21.672464 18876 leader_election.cc:302] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bbb399e30fd43509d2e7d355ededfc6, b9b9d3dc329945d69d75197d1c37dce2; no voters:
I20260927 00:45:21.672639 19013 raft_consensus.cc:2781] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:21.672866 19013 raft_consensus.cc:686] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 1 LEADER]: Becoming Leader. State: Replica: 7bbb399e30fd43509d2e7d355ededfc6, State: Running, Role: LEADER
I20260927 00:45:21.673225 19013 consensus_queue.cc:237] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } }
I20260927 00:45:21.676175 18529 catalog_manager.cc:5495] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bbb399e30fd43509d2e7d355ededfc6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7bbb399e30fd43509d2e7d355ededfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bbb399e30fd43509d2e7d355ededfc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 } health_report { overall_health: UNKNOWN } } }
I20260927 00:45:21.704209 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.700536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.0.98.3:36285)
0927 00:45:21.700682 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.700687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.700702 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:45:21.700848 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.700851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.700966 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.701089 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:21.701338 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.701842 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.702716 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.702937 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.703046 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.703369 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:21.703774 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:21.703939 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.704001 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.704039 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 00:45:21.709416 18989 tablet.cc:2369] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:21.709971 18674 raft_consensus.cc:1261] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2 [term 1 FOLLOWER]: Refusing update from remote peer 7bbb399e30fd43509d2e7d355ededfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:21.709986 18808 raft_consensus.cc:1261] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 1 FOLLOWER]: Refusing update from remote peer 7bbb399e30fd43509d2e7d355ededfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:21.710686 19015 consensus_queue.cc:1035] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b9d3dc329945d69d75197d1c37dce2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:21.710863 19013 consensus_queue.cc:1035] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d4dcb49bdd4a2e8f0a8c7c402547db" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:21.725510 18529 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:21.725680 18529 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:21.727957 18529 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fec66d96fd0427281c17b7d13689b23: Sending DeleteTablet for 3 replicas of tablet 43bac2896d814a50b4ce41f8edaad01d
I20260927 00:45:21.728668 18922 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bac2896d814a50b4ce41f8edaad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47180
I20260927 00:45:21.728830 18788 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bac2896d814a50b4ce41f8edaad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54862
I20260927 00:45:21.729302 18654 tablet_service.cc:1514] Processing DeleteTablet for tablet 43bac2896d814a50b4ce41f8edaad01d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42356
I20260927 00:45:21.729771 19030 tablet_replica.cc:330] T 43bac2896d814a50b4ce41f8edaad01d P b9b9d3dc329945d69d75197d1c37dce2: stopping tablet replica
I20260927 00:45:21.729851 19029 tablet_replica.cc:330] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: stopping tablet replica
I20260927 00:45:21.730062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18591
I20260927 00:45:21.730181 19029 raft_consensus.cc:2229] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:21.730459 19029 raft_consensus.cc:2258] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:21.732513 19029 ts_tablet_manager.cc:1905] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:21.735180 19029 ts_tablet_manager.cc:1918] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:21.735273 19029 log.cc:1196] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/43bac2896d814a50b4ce41f8edaad01d
I20260927 00:45:21.735625 19029 ts_tablet_manager.cc:1939] T 43bac2896d814a50b4ce41f8edaad01d P 74d4dcb49bdd4a2e8f0a8c7c402547db: Deleting consensus metadata
I20260927 00:45:21.729598 19028 tablet_replica.cc:330] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: stopping tablet replica
I20260927 00:45:21.736840 18514 catalog_manager.cc:4841] TS 74d4dcb49bdd4a2e8f0a8c7c402547db (127.0.98.2:38369): tablet 43bac2896d814a50b4ce41f8edaad01d (table test-table [id=6dce48de6cc947b3bd35b71b08dded82]) successfully deleted
I20260927 00:45:21.737035 19028 raft_consensus.cc:2229] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:21.737388 19028 raft_consensus.cc:2258] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 00:45:21.737703 18514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39315 (error 108)
I20260927 00:45:21.737860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18725
I20260927 00:45:21.739234 19028 ts_tablet_manager.cc:1905] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:21.741717 19028 ts_tablet_manager.cc:1918] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:21.741842 19028 log.cc:1196] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/43bac2896d814a50b4ce41f8edaad01d
I20260927 00:45:21.742189 19028 ts_tablet_manager.cc:1939] T 43bac2896d814a50b4ce41f8edaad01d P 7bbb399e30fd43509d2e7d355ededfc6: Deleting consensus metadata
I20260927 00:45:21.742401 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.742019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39315 (local address 127.0.0.1:42364)
0927 00:45:21.742224 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:21.742297 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 00:45:21.742652 18514 catalog_manager.cc:4584] TS b9b9d3dc329945d69d75197d1c37dce2 (127.0.98.1:39315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43bac2896d814a50b4ce41f8edaad01d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39315: connect: Connection refused (error 111)
I20260927 00:45:21.743624 18514 catalog_manager.cc:4841] TS 7bbb399e30fd43509d2e7d355ededfc6 (127.0.98.3:36285): tablet 43bac2896d814a50b4ce41f8edaad01d (table test-table [id=6dce48de6cc947b3bd35b71b08dded82]) successfully deleted
I20260927 00:45:21.744284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18859
I20260927 00:45:21.749315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 18497
2026-09-27T00:45:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2320 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T00:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:21Z Disabled control of system clock
I20260927 00:45:21.778414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:21.888622 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:21.888865 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:21.888921 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:21.892717 19035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:21.892762 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:21.892778 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:21.892794 19035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:21.892809 19035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:21.897472 19035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:21.898386 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:21.899642 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:21.907459 19040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:21.907473 19043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:21.907471 19041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:21.907528 19035 server_base.cc:1034] running on GCE node
I20260927 00:45:21.908255 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:21.909401 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:21.910559 19035 hybrid_clock.cc:648] HybridClock initialized: now 1790469921910540 us; error 32 us; skew 500 ppm
I20260927 00:45:21.912420 19035 webserver.cc:460] Webserver started at http://127.0.98.62:33045/ using document root <none> and password file <none>
I20260927 00:45:21.913038 19035 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:21.913105 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:21.913326 19035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:21.915263 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "5410a378a3b045698b4e1232b406670c"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "c43ddf94677e17df4907bfd33a06303c"
server_key_iv: "81460716a0bfe99270772d434593236d"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.915820 19035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "5410a378a3b045698b4e1232b406670c"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "c43ddf94677e17df4907bfd33a06303c"
server_key_iv: "81460716a0bfe99270772d434593236d"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.919340 19035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:21.921720 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:21.922886 19035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:21.923025 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "5410a378a3b045698b4e1232b406670c"
format_stamp: "Formatted at 2026-09-27 00:45:21 on dist-test-slave-pkws"
server_key: "c43ddf94677e17df4907bfd33a06303c"
server_key_iv: "81460716a0bfe99270772d434593236d"
server_key_version: "encryptionkey@0"
I20260927 00:45:21.923136 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:21.936983 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:21.937577 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:21.937764 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:21.944382 19035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41421
I20260927 00:45:21.944394 19101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41421 every 8 connection(s)
I20260927 00:45:21.945303 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:21.948129 19102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:21.953162 19102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: Bootstrap starting.
I20260927 00:45:21.953505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19035
I20260927 00:45:21.953635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:21.953933 392 external_mini_cluster.cc:1428] Setting key ee17f5be4d543df5632d95f9102c1a16
I20260927 00:45:21.955724 19102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:21.956446 19102 log.cc:824] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:21.958210 19102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: No bootstrap required, opened a new log
I20260927 00:45:21.961531 19102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 00:45:21.961716 19102 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:21.961771 19102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5410a378a3b045698b4e1232b406670c, State: Initialized, Role: FOLLOWER
I20260927 00:45:21.962225 19102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 00:45:21.962345 19102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:21.962427 19102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:21.962533 19102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:21.963127 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:21.955227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.0.98.62:41421)
0927 00:45:21.955862 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:21.955876 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:21.955910 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:45:21.956546 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:21.956578 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:21.957070 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:21.957432 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:21.957840 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.958817 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.959700 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:21.960163 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:21.960329 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:21.961237 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:21.961242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:21.961253 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:21.961623 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:21.961673 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:21.961692 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:21.961878 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:21.962060 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 00:45:21.962342 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260927 00:45:21.963349 19102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 00:45:21.963661 19102 leader_election.cc:302] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5410a378a3b045698b4e1232b406670c; no voters:
I20260927 00:45:21.963953 19102 leader_election.cc:288] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:21.964015 19107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:21.964246 19107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [term 1 LEADER]: Becoming Leader. State: Replica: 5410a378a3b045698b4e1232b406670c, State: Running, Role: LEADER
I20260927 00:45:21.964572 19107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260927 00:45:21.965195 19102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:21.966167 19108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5410a378a3b045698b4e1232b406670c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260927 00:45:21.966287 19108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:21.966480 19109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5410a378a3b045698b4e1232b406670c. Latest consensus state: current_term: 1 leader_uuid: "5410a378a3b045698b4e1232b406670c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5410a378a3b045698b4e1232b406670c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260927 00:45:21.966676 19109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:21.966904 19116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:21.969995 19116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:21.975462 19116 catalog_manager.cc:1347] Generated new cluster ID: 2eac34bd6e5f4efb9bbec533347896a8
I20260927 00:45:21.975523 19116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:21.984169 19116 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:21.985296 19116 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:21.995324 19116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: Generated new TSK 0
I20260927 00:45:21.995929 19116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:22.003782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--builtin_ntp_servers=127.0.98.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:22.121981 19126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:22.122213 19126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:22.122249 19126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:22.126233 19126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:22.126284 19126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:22.126391 19126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:22.131165 19126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:22.132189 19126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:22.133540 19126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:22.142951 19131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:22.142980 19134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:22.142980 19132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:22.143296 19126 server_base.cc:1034] running on GCE node
I20260927 00:45:22.143704 19126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:22.144331 19126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:22.145509 19126 hybrid_clock.cc:648] HybridClock initialized: now 1790469922145471 us; error 51 us; skew 500 ppm
I20260927 00:45:22.147228 19126 webserver.cc:460] Webserver started at http://127.0.98.1:33909/ using document root <none> and password file <none>
I20260927 00:45:22.147790 19126 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:22.147882 19126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:22.148111 19126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:22.150162 19126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "f55da6dcd1f04b6382acc768ebf9a39c"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "15815990779fcc82921d9b0a60f651d1"
server_key_iv: "345bb4c279964aa73858881cee24b5b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.150702 19126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f55da6dcd1f04b6382acc768ebf9a39c"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "15815990779fcc82921d9b0a60f651d1"
server_key_iv: "345bb4c279964aa73858881cee24b5b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.154212 19126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:22.156422 19140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.157600 19126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:22.157737 19126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f55da6dcd1f04b6382acc768ebf9a39c"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "15815990779fcc82921d9b0a60f651d1"
server_key_iv: "345bb4c279964aa73858881cee24b5b1"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.157841 19126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:22.191449 19126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:22.192133 19126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:22.192330 19126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:22.192946 19126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:22.193861 19126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:22.193936 19126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.194007 19126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:22.194059 19126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.202751 19126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45827
I20260927 00:45:22.202783 19253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45827 every 8 connection(s)
I20260927 00:45:22.203611 19126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:22.210621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19126
I20260927 00:45:22.210788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:22.211063 392 external_mini_cluster.cc:1428] Setting key 3fab73ba5db5e6a8b837b1204adc7bfb
I20260927 00:45:22.212723 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.205375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43001 (local address 127.0.98.62:41421)
0927 00:45:22.205505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.205509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.206520 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 00:45:22.207971 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.207975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.208031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.208141 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:22.209047 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.209591 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.210715 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.211004 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.211124 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.212176 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:22.212182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:22.212185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:22.212198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:22.212219 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:22.212229 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:22.212359 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.212523 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.212585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:45:22.213047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--builtin_ntp_servers=127.0.98.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:22.213698 19256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.205757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:43001)
0927 00:45:22.206378 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.206417 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.207758 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.208309 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.208316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.208395 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:22.208854 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.208866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.209716 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.209719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.210558 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.210576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.211158 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.211162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.211326 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.211930 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:22.211931 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:22.211969 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:22.212025 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:22.212364 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:22.212377 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:22.212384 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.212521 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.212800 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260927 00:45:22.215493 19254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 00:45:22.215727 19254 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:22.216200 19254 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 00:45:22.217633 19066 ts_manager.cc:194] Registered new tserver with Master: f55da6dcd1f04b6382acc768ebf9a39c (127.0.98.1:45827)
I20260927 00:45:22.218737 19066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43001
W20260927 00:45:22.323951 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:22.324219 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:22.324297 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:22.328500 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:22.328567 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:22.328648 19257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:22.333316 19257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:22.334280 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:22.335592 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:22.345079 19257 server_base.cc:1034] running on GCE node
W20260927 00:45:22.345053 19265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:22.345103 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:22.345294 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:22.345678 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:22.346313 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:22.347478 19257 hybrid_clock.cc:648] HybridClock initialized: now 1790469922347470 us; error 31 us; skew 500 ppm
I20260927 00:45:22.349193 19257 webserver.cc:460] Webserver started at http://127.0.98.2:35731/ using document root <none> and password file <none>
I20260927 00:45:22.349727 19257 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:22.349786 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:22.350035 19257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:22.351920 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "0e16eb0cc92747519b55a2c2f7b3c66b"
server_key_iv: "745d0eb9e636095855817bf912d4d55a"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.352450 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "0e16eb0cc92747519b55a2c2f7b3c66b"
server_key_iv: "745d0eb9e636095855817bf912d4d55a"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.356007 19257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:45:22.358284 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.359392 19257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:22.359524 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "0e16eb0cc92747519b55a2c2f7b3c66b"
server_key_iv: "745d0eb9e636095855817bf912d4d55a"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.359629 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:22.383057 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:22.383728 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:22.383926 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:22.384495 19257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:22.385429 19257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:22.385502 19257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.385572 19257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:22.385623 19257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.394160 19257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36341
I20260927 00:45:22.394186 19384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36341 every 8 connection(s)
I20260927 00:45:22.395056 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:22.399245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19257
I20260927 00:45:22.399344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:22.399645 392 external_mini_cluster.cc:1428] Setting key 243cc126e30d6d7bb17f88e8dd99ec41
I20260927 00:45:22.401558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--builtin_ntp_servers=127.0.98.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:22.404594 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.396959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60431 (local address 127.0.98.62:41421)
0927 00:45:22.397107 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.397111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.398185 (+ 1074us) server_negotiation.cc:407] Connection header received
0927 00:45:22.399595 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.399598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.399660 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.399739 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:22.400721 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.401348 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.402598 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.402835 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.402950 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.404142 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:22.404145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:22.404147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:22.404158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:22.404178 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:22.404186 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:22.404242 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.404437 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.404486 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:45:22.405835 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.397371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:60431)
0927 00:45:22.398023 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.398072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.399336 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.399910 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.399916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.399999 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:22.400577 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.400591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.401480 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.401488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.402464 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.402473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.402987 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.402991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.403161 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.403834 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:22.403838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:22.403895 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:22.403977 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:22.404272 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:22.404283 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:22.404292 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.404423 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.404735 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260927 00:45:22.407060 19385 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 00:45:22.407285 19385 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:22.407770 19385 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 00:45:22.408807 19066 ts_manager.cc:194] Registered new tserver with Master: d1f9828d1d5a4adcbac86e8b87fd85e4 (127.0.98.2:36341)
I20260927 00:45:22.409425 19066 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60431
W20260927 00:45:22.510269 19388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:22.510552 19388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:22.510653 19388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:22.515036 19388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:22.515107 19388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:22.515189 19388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:22.519840 19388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:22.520783 19388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:22.522068 19388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:22.531139 19394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:22.531185 19396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:22.531277 19388 server_base.cc:1034] running on GCE node
W20260927 00:45:22.531190 19393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:22.531810 19388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:22.532331 19388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:22.533478 19388 hybrid_clock.cc:648] HybridClock initialized: now 1790469922533462 us; error 29 us; skew 500 ppm
I20260927 00:45:22.535084 19388 webserver.cc:460] Webserver started at http://127.0.98.3:41187/ using document root <none> and password file <none>
I20260927 00:45:22.535567 19388 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:22.535622 19388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:22.535770 19388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:22.537758 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "94526c65bec74d90ba8d59d753bbab6e"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "939e1c1385e187e5c3d3900ed849bdb0"
server_key_iv: "b5efa06fd26e31a502a0c54a21bb163e"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.538249 19388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94526c65bec74d90ba8d59d753bbab6e"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "939e1c1385e187e5c3d3900ed849bdb0"
server_key_iv: "b5efa06fd26e31a502a0c54a21bb163e"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.541742 19388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:22.543926 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.545115 19388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:22.545240 19388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "94526c65bec74d90ba8d59d753bbab6e"
format_stamp: "Formatted at 2026-09-27 00:45:22 on dist-test-slave-pkws"
server_key: "939e1c1385e187e5c3d3900ed849bdb0"
server_key_iv: "b5efa06fd26e31a502a0c54a21bb163e"
server_key_version: "encryptionkey@0"
I20260927 00:45:22.545346 19388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:22.556268 19388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:22.556880 19388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:22.557130 19388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:22.557739 19388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:22.558616 19388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:22.558684 19388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.558750 19388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:22.558802 19388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:22.567871 19388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43817
I20260927 00:45:22.567902 19515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43817 every 8 connection(s)
I20260927 00:45:22.568720 19388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:22.577589 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.570450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44251 (local address 127.0.98.62:41421)
0927 00:45:22.570581 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.570584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.571555 (+ 971us) server_negotiation.cc:407] Connection header received
0927 00:45:22.572968 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.572972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.573038 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.573132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:22.574008 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.574539 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.575634 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.575854 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.575991 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.577094 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:22.577098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:22.577114 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:22.577131 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:22.577150 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:22.577158 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:22.577233 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.577389 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.577449 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:22.577977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19388
I20260927 00:45:22.578084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:22.578369 392 external_mini_cluster.cc:1428] Setting key b9b43639afcbadcfe9f9ba24f263979a
I20260927 00:45:22.578596 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.570784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:44251)
0927 00:45:22.571384 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.571444 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.572710 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.573300 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.573306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.573384 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:22.573855 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.573872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.574667 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.574675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.575510 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.575518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.576024 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.576028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.576188 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.576811 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:22.576812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:22.576850 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:22.576930 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:22.577247 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:22.577257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:22.577268 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.577395 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.577653 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260927 00:45:22.579777 19516 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260927 00:45:22.579988 19516 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:22.580453 19516 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260927 00:45:22.581404 19065 ts_manager.cc:194] Registered new tserver with Master: 94526c65bec74d90ba8d59d753bbab6e (127.0.98.3:43817)
I20260927 00:45:22.581990 19065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44251
I20260927 00:45:22.592155 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:22.597579 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.594070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.0.98.62:41421)
0927 00:45:22.594183 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.594186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.594319 (+ 133us) server_negotiation.cc:407] Connection header received
0927 00:45:22.594408 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.594410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.594445 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.594509 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:22.594739 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.595230 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.595994 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.596198 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.596286 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.597200 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:22.597203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:22.597204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:22.597214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:22.597231 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:22.597238 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:22.597327 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.597449 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.597480 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:22.600360 19065 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:22.603272 19065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:22.617012 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.611782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.98.2:36341)
0927 00:45:22.612209 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.612221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.612243 (+ 22us) server_negotiation.cc:407] Connection header received
0927 00:45:22.612557 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.612562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.612750 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.613179 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:22.613326 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614109 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.615666 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616293 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.616432 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.616632 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.616731 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.616829 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":129,"threads_started":1}
I20260927 00:45:22.617005 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.611107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36341 (local address 127.0.0.1:49990)
0927 00:45:22.611815 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.611830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.611921 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.612926 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.612933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.612949 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:22.613039 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.613045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614226 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.614230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.615508 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.615520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616432 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.616436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.616553 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.616595 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.616708 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.616836 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":137,"thread_start_us":110,"threads_started":1}
I20260927 00:45:22.617185 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.611811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43817 (local address 127.0.0.1:58196)
0927 00:45:22.612378 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.612391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.612460 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.613275 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.613277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.613293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:22.613384 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.613390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614690 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.614694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.615788 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.615796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616799 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.616805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.616880 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.616926 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.616975 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.617041 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":102,"threads_started":1}
I20260927 00:45:22.617304 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.611004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45827 (local address 127.0.0.1:35914)
0927 00:45:22.611311 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.611353 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.611473 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.612351 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.612358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.612379 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:22.612572 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.612579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614188 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.614194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.615504 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.615515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616565 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.616569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.616666 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.616695 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.616746 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.617204 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":400,"thread_start_us":98,"threads_started":1}
I20260927 00:45:22.617601 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.611110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.0.98.1:45827)
0927 00:45:22.611836 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.611851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.611878 (+ 27us) server_negotiation.cc:407] Connection header received
0927 00:45:22.611959 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.611968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.612170 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.612728 (+ 558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:22.612870 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614066 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.615638 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616423 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.616837 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.617053 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.617350 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.617441 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":135,"threads_started":1}
I20260927 00:45:22.617838 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.612241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.98.3:43817)
0927 00:45:22.612826 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.612839 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.612862 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:45:22.612957 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.612962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.613135 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.613284 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:22.613511 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.614574 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.615912 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.616671 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.617360 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.617536 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.617583 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.617678 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":131,"threads_started":1}
I20260927 00:45:22.620232 19188 tablet_service.cc:1467] Processing CreateTablet for tablet 3691e807661c4512891d94a59cfbd718 (DEFAULT_TABLE table=test-table [id=0cedfdc9f95e4241a5b23a4d3ed8e22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:22.620319 19319 tablet_service.cc:1467] Processing CreateTablet for tablet 3691e807661c4512891d94a59cfbd718 (DEFAULT_TABLE table=test-table [id=0cedfdc9f95e4241a5b23a4d3ed8e22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:22.620319 19450 tablet_service.cc:1467] Processing CreateTablet for tablet 3691e807661c4512891d94a59cfbd718 (DEFAULT_TABLE table=test-table [id=0cedfdc9f95e4241a5b23a4d3ed8e22c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:22.621253 19188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3691e807661c4512891d94a59cfbd718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:22.621294 19319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3691e807661c4512891d94a59cfbd718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:22.621294 19450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3691e807661c4512891d94a59cfbd718. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:22.627759 19533 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Bootstrap starting.
I20260927 00:45:22.628643 19534 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Bootstrap starting.
I20260927 00:45:22.628803 19535 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Bootstrap starting.
I20260927 00:45:22.629804 19533 tablet_bootstrap.cc:654] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:22.630511 19533 log.cc:824] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:22.630715 19535 tablet_bootstrap.cc:654] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:22.630714 19534 tablet_bootstrap.cc:654] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:22.631531 19535 log.cc:824] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:22.631531 19534 log.cc:824] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:22.632423 19533 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: No bootstrap required, opened a new log
I20260927 00:45:22.632624 19533 ts_tablet_manager.cc:1397] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:45:22.633272 19535 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: No bootstrap required, opened a new log
I20260927 00:45:22.633476 19535 ts_tablet_manager.cc:1397] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:22.633790 19534 tablet_bootstrap.cc:492] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: No bootstrap required, opened a new log
I20260927 00:45:22.634017 19534 ts_tablet_manager.cc:1397] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:45:22.636471 19535 raft_consensus.cc:348] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.636687 19535 raft_consensus.cc:374] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:22.636758 19535 raft_consensus.cc:729] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f9828d1d5a4adcbac86e8b87fd85e4, State: Initialized, Role: FOLLOWER
I20260927 00:45:22.636953 19533 raft_consensus.cc:348] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.637200 19533 raft_consensus.cc:374] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:22.637291 19533 raft_consensus.cc:729] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f55da6dcd1f04b6382acc768ebf9a39c, State: Initialized, Role: FOLLOWER
I20260927 00:45:22.637239 19535 consensus_queue.cc:260] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.637699 19533 consensus_queue.cc:260] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.638129 19535 ts_tablet_manager.cc:1428] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:22.638336 19254 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260927 00:45:22.638478 19385 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260927 00:45:22.638574 19533 ts_tablet_manager.cc:1428] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 00:45:22.638088 19534 raft_consensus.cc:348] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.639323 19534 raft_consensus.cc:374] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:22.639415 19534 raft_consensus.cc:729] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94526c65bec74d90ba8d59d753bbab6e, State: Initialized, Role: FOLLOWER
I20260927 00:45:22.639822 19534 consensus_queue.cc:260] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.640694 19516 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260927 00:45:22.641010 19534 ts_tablet_manager.cc:1428] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
W20260927 00:45:22.645874 19386 tablet.cc:2369] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:45:22.704703 19255 tablet.cc:2369] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:45:22.819657 19517 tablet.cc:2369] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:22.843570 19541 raft_consensus.cc:482] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:22.843745 19541 raft_consensus.cc:504] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.844828 19541 leader_election.cc:288] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f55da6dcd1f04b6382acc768ebf9a39c (127.0.98.1:45827), d1f9828d1d5a4adcbac86e8b87fd85e4 (127.0.98.2:36341)
I20260927 00:45:22.849645 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.845263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58533 (local address 127.0.98.2:36341)
0927 00:45:22.845407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.845412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.845426 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:22.845481 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.845486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.845540 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.845606 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:22.845947 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.846710 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.848348 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.848990 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.849093 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.849151 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.849449 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.849519 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:45:22.849645 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.845139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42701 (local address 127.0.98.1:45827)
0927 00:45:22.845280 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.845284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.845685 (+ 401us) server_negotiation.cc:407] Connection header received
0927 00:45:22.845765 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.845767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.845808 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.845866 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:22.846124 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.846822 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.848348 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.848974 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.849092 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.849150 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.849454 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.849519 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:45:22.849756 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.845073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36341 (local address 127.0.98.3:58533)
0927 00:45:22.845214 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.845231 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.845321 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.845649 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.845651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.845670 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:22.845805 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.845811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.846814 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.846816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.848230 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.848241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.849109 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.849113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.849305 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.849343 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.849420 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.849527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":162}
I20260927 00:45:22.849766 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.845070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45827 (local address 127.0.98.3:42701)
0927 00:45:22.845561 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.845571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:22.845685 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:22.845898 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:22.845902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:22.845921 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:22.846017 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.846021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.846913 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.846916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.848231 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:22.848241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.849118 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.849121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:22.849299 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.849346 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 00:45:22.849416 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 00:45:22.849527 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":245,"spinlock_wait_cycles":2944,"thread_start_us":150,"threads_started":1}
I20260927 00:45:22.850382 19208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3691e807661c4512891d94a59cfbd718" candidate_uuid: "94526c65bec74d90ba8d59d753bbab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" is_pre_election: true
I20260927 00:45:22.850589 19339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3691e807661c4512891d94a59cfbd718" candidate_uuid: "94526c65bec74d90ba8d59d753bbab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" is_pre_election: true
I20260927 00:45:22.850692 19208 raft_consensus.cc:2454] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94526c65bec74d90ba8d59d753bbab6e in term 0.
I20260927 00:45:22.850879 19339 raft_consensus.cc:2454] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94526c65bec74d90ba8d59d753bbab6e in term 0.
I20260927 00:45:22.851183 19404 leader_election.cc:302] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94526c65bec74d90ba8d59d753bbab6e, f55da6dcd1f04b6382acc768ebf9a39c; no voters:
I20260927 00:45:22.851475 19541 raft_consensus.cc:2781] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:22.851574 19541 raft_consensus.cc:482] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:22.851630 19541 raft_consensus.cc:3037] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:22.852579 19541 raft_consensus.cc:504] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.852984 19541 leader_election.cc:288] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [CANDIDATE]: Term 1 election: Requested vote from peers f55da6dcd1f04b6382acc768ebf9a39c (127.0.98.1:45827), d1f9828d1d5a4adcbac86e8b87fd85e4 (127.0.98.2:36341)
I20260927 00:45:22.853281 19208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3691e807661c4512891d94a59cfbd718" candidate_uuid: "94526c65bec74d90ba8d59d753bbab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f55da6dcd1f04b6382acc768ebf9a39c"
I20260927 00:45:22.853302 19339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3691e807661c4512891d94a59cfbd718" candidate_uuid: "94526c65bec74d90ba8d59d753bbab6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4"
I20260927 00:45:22.853426 19339 raft_consensus.cc:3037] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:22.853410 19208 raft_consensus.cc:3037] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:22.854637 19208 raft_consensus.cc:2454] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94526c65bec74d90ba8d59d753bbab6e in term 1.
I20260927 00:45:22.854637 19339 raft_consensus.cc:2454] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94526c65bec74d90ba8d59d753bbab6e in term 1.
I20260927 00:45:22.854929 19405 leader_election.cc:302] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94526c65bec74d90ba8d59d753bbab6e, d1f9828d1d5a4adcbac86e8b87fd85e4; no voters:
I20260927 00:45:22.855121 19541 raft_consensus.cc:2781] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:22.855347 19541 raft_consensus.cc:686] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 1 LEADER]: Becoming Leader. State: Replica: 94526c65bec74d90ba8d59d753bbab6e, State: Running, Role: LEADER
I20260927 00:45:22.855700 19541 consensus_queue.cc:237] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } }
I20260927 00:45:22.858704 19066 catalog_manager.cc:5495] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 94526c65bec74d90ba8d59d753bbab6e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "94526c65bec74d90ba8d59d753bbab6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94526c65bec74d90ba8d59d753bbab6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43817 } health_report { overall_health: HEALTHY } } }
I20260927 00:45:22.907830 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.904397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.98.3:43817)
0927 00:45:22.904526 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:22.904530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:22.904543 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:22.904615 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:22.904617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:22.904672 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:22.904763 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:22.904991 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.905484 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.906301 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:22.906501 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:22.906638 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:22.906953 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:22.907356 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:22.907520 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:22.907649 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 00:45:22.907684 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:22.913525 19339 raft_consensus.cc:1261] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 1 FOLLOWER]: Refusing update from remote peer 94526c65bec74d90ba8d59d753bbab6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:22.913551 19208 raft_consensus.cc:1261] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 1 FOLLOWER]: Refusing update from remote peer 94526c65bec74d90ba8d59d753bbab6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:22.914248 19541 consensus_queue.cc:1035] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f9828d1d5a4adcbac86e8b87fd85e4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:22.914464 19543 consensus_queue.cc:1035] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f55da6dcd1f04b6382acc768ebf9a39c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:22.930795 19065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:22.930928 19065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:22.933089 19065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5410a378a3b045698b4e1232b406670c: Sending DeleteTablet for 3 replicas of tablet 3691e807661c4512891d94a59cfbd718
I20260927 00:45:22.933804 19188 tablet_service.cc:1514] Processing DeleteTablet for tablet 3691e807661c4512891d94a59cfbd718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:22 UTC) from {username='slave'} at 127.0.0.1:35914
I20260927 00:45:22.933954 19319 tablet_service.cc:1514] Processing DeleteTablet for tablet 3691e807661c4512891d94a59cfbd718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:22 UTC) from {username='slave'} at 127.0.0.1:49990
I20260927 00:45:22.934082 19450 tablet_service.cc:1514] Processing DeleteTablet for tablet 3691e807661c4512891d94a59cfbd718 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:22 UTC) from {username='slave'} at 127.0.0.1:58196
I20260927 00:45:22.934429 19556 tablet_replica.cc:330] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c: stopping tablet replica
I20260927 00:45:22.934523 19558 tablet_replica.cc:330] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: stopping tablet replica
I20260927 00:45:22.934775 19558 raft_consensus.cc:2229] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:22.934775 19556 raft_consensus.cc:2229] T 3691e807661c4512891d94a59cfbd718 P f55da6dcd1f04b6382acc768ebf9a39c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:22.934932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19126
I20260927 00:45:22.935889 19557 tablet_replica.cc:330] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: stopping tablet replica
I20260927 00:45:22.936177 19557 raft_consensus.cc:2229] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:22.936513 19557 raft_consensus.cc:2258] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:22.938058 19557 ts_tablet_manager.cc:1905] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:22.938192 19558 raft_consensus.cc:2258] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:22.939550 19558 ts_tablet_manager.cc:1905] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:22.941411 19557 ts_tablet_manager.cc:1918] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:22.941573 19557 log.cc:1196] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/3691e807661c4512891d94a59cfbd718
I20260927 00:45:22.942121 19558 ts_tablet_manager.cc:1918] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:22.942183 19557 ts_tablet_manager.cc:1939] T 3691e807661c4512891d94a59cfbd718 P d1f9828d1d5a4adcbac86e8b87fd85e4: Deleting consensus metadata
I20260927 00:45:22.942274 19558 log.cc:1196] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/3691e807661c4512891d94a59cfbd718
I20260927 00:45:22.942610 19558 ts_tablet_manager.cc:1939] T 3691e807661c4512891d94a59cfbd718 P 94526c65bec74d90ba8d59d753bbab6e: Deleting consensus metadata
W20260927 00:45:22.943064 19051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45827 (error 108)
I20260927 00:45:22.943614 19052 catalog_manager.cc:4841] TS d1f9828d1d5a4adcbac86e8b87fd85e4 (127.0.98.2:36341): tablet 3691e807661c4512891d94a59cfbd718 (table test-table [id=0cedfdc9f95e4241a5b23a4d3ed8e22c]) successfully deleted
I20260927 00:45:22.943794 19052 catalog_manager.cc:4841] TS 94526c65bec74d90ba8d59d753bbab6e (127.0.98.3:43817): tablet 3691e807661c4512891d94a59cfbd718 (table test-table [id=0cedfdc9f95e4241a5b23a4d3ed8e22c]) successfully deleted
I20260927 00:45:22.943982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19257
I20260927 00:45:22.944716 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:22.944092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45827 (local address 127.0.0.1:35930)
0927 00:45:22.944553 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:22.944624 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
W20260927 00:45:22.949107 19051 catalog_manager.cc:4584] TS f55da6dcd1f04b6382acc768ebf9a39c (127.0.98.1:45827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3691e807661c4512891d94a59cfbd718: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45827: connect: Connection refused (error 111)
I20260927 00:45:22.950555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19388
I20260927 00:45:22.956401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19035
2026-09-27T00:45:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:23 dist-test-slave-pkws krb5kdc[19562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:23 dist-test-slave-pkws krb5kdc[19562](info): set up 2 sockets
Sep 27 00:45:23 dist-test-slave-pkws krb5kdc[19562](info): commencing operation
krb5kdc: starting...
W20260927 00:45:23.996811 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:24.117340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:24.223510 19575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:24.223769 19575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:24.223853 19575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:24.227742 19575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:24.227823 19575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:24.227862 19575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:24.227919 19575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:24.227960 19575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:24.233000 19575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:24.233893 19575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:24.234798 19575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:24.242811 19575 server_base.cc:1034] running on GCE node
W20260927 00:45:24.242821 19581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.242821 19583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.242952 19580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.243649 19575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:24.244652 19575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:24.245834 19575 hybrid_clock.cc:648] HybridClock initialized: now 1790469924245816 us; error 41 us; skew 500 ppm
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:24.248348 19575 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:24.249243 19575 webserver.cc:460] Webserver started at http://127.0.98.62:37219/ using document root <none> and password file <none>
I20260927 00:45:24.249764 19575 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:24.249866 19575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:24.250115 19575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:24.252010 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "40feddb2077d4a09a412358f8240f144"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "020657dff0a40baa50058a87b33de070"
server_key_iv: "d5dbb456e6151fa431b0719a2222a2b0"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.252569 19575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "40feddb2077d4a09a412358f8240f144"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "020657dff0a40baa50058a87b33de070"
server_key_iv: "d5dbb456e6151fa431b0719a2222a2b0"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.256100 19575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:24.258361 19590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.259516 19575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:24.259650 19575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "40feddb2077d4a09a412358f8240f144"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "020657dff0a40baa50058a87b33de070"
server_key_iv: "d5dbb456e6151fa431b0719a2222a2b0"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.259758 19575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:24.273830 19575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:24.276459 19575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:24.276660 19575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:24.283466 19575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44193
I20260927 00:45:24.283493 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44193 every 8 connection(s)
I20260927 00:45:24.284346 19575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:24.286873 19643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:24.292016 19643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: Bootstrap starting.
I20260927 00:45:24.292514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19575
I20260927 00:45:24.292650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:24.292994 392 external_mini_cluster.cc:1428] Setting key 282c7df5da8e21807a2fa0ad9917ca5a
I20260927 00:45:24.294775 19643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:24.295470 19643 log.cc:824] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:24.297559 19643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: No bootstrap required, opened a new log
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:24.301913 19643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260927 00:45:24.302131 19643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:24.302188 19643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40feddb2077d4a09a412358f8240f144, State: Initialized, Role: FOLLOWER
I20260927 00:45:24.302632 19643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260927 00:45:24.302775 19643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:24.302858 19643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:24.302974 19643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:24.304020 19643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260927 00:45:24.304345 19643 leader_election.cc:302] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40feddb2077d4a09a412358f8240f144; no voters:
I20260927 00:45:24.304697 19643 leader_election.cc:288] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:24.304734 19648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:24.304944 19648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [term 1 LEADER]: Becoming Leader. State: Replica: 40feddb2077d4a09a412358f8240f144, State: Running, Role: LEADER
I20260927 00:45:24.305277 19648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260927 00:45:24.305926 19643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:24.306387 19650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40feddb2077d4a09a412358f8240f144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } } }
I20260927 00:45:24.306560 19650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:24.307626 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.294281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.0.98.62:44193)
0927 00:45:24.294683 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:24.294697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:24.294731 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:45:24.295382 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:24.295398 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:24.295780 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:24.296162 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:24.297037 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.298034 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.299203 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.299652 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.299762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:24.302712 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:24.302716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:24.302728 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:24.302753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:24.304549 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.305089 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.305094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.305099 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.305228 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.305524 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.305538 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.305542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.305911 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:24.306127 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:24.306450 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 00:45:24.306760 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260927 00:45:24.308761 19649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40feddb2077d4a09a412358f8240f144. Latest consensus state: current_term: 1 leader_uuid: "40feddb2077d4a09a412358f8240f144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40feddb2077d4a09a412358f8240f144" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } } }
I20260927 00:45:24.308917 19649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144 [sys.catalog]: This master's current role is: LEADER
W20260927 00:45:24.311427 19664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:24.311515 19664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:24.311614 19652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:24.312489 19652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:24.317538 19652 catalog_manager.cc:1347] Generated new cluster ID: 5a46fa9c05424ca69912dc84160b65d0
I20260927 00:45:24.317601 19652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:24.339110 19652 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:24.340003 19652 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:24.350826 19652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: Generated new TSK 0
I20260927 00:45:24.351449 19652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 00:45:24.379364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:24.513118 19669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:24.513334 19669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:24.513373 19669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:24.517437 19669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:24.517485 19669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:24.517588 19669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:24.522493 19669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:24.523396 19669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:24.524320 19669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:24.533493 19674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.533493 19677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.533524 19669 server_base.cc:1034] running on GCE node
W20260927 00:45:24.533540 19675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.534045 19669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:24.534664 19669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:24.535847 19669 hybrid_clock.cc:648] HybridClock initialized: now 1790469924535822 us; error 37 us; skew 500 ppm
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:24.538450 19669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 00:45:24.539256 19669 webserver.cc:460] Webserver started at http://127.0.98.1:34429/ using document root <none> and password file <none>
I20260927 00:45:24.539773 19669 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:24.539877 19669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:24.540092 19669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:24.542014 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "e24ffcf91b644d979c5d0f72df302bb8"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3f29dd6857fe619fb4820be9d6b8a6ab"
server_key_iv: "640a1b929280f79bfc94eca9fe800c5a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.542562 19669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e24ffcf91b644d979c5d0f72df302bb8"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3f29dd6857fe619fb4820be9d6b8a6ab"
server_key_iv: "640a1b929280f79bfc94eca9fe800c5a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.546043 19669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:24.548301 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.549467 19669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:24.549595 19669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "e24ffcf91b644d979c5d0f72df302bb8"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3f29dd6857fe619fb4820be9d6b8a6ab"
server_key_iv: "640a1b929280f79bfc94eca9fe800c5a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.549702 19669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:24.564002 19669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:24.566697 19669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:24.566896 19669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:24.567433 19669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:24.568222 19669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:24.568291 19669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.568354 19669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:24.568406 19669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.576807 19669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39317
I20260927 00:45:24.576841 19797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39317 every 8 connection(s)
I20260927 00:45:24.577631 19669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:24.585564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19669
I20260927 00:45:24.585705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:24.586030 392 external_mini_cluster.cc:1428] Setting key 1503f7427dd44bb59ea821c3fc928c81
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:24.590593 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.579297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45671 (local address 127.0.98.62:44193)
0927 00:45:24.579430 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:24.579434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:24.580188 (+ 754us) server_negotiation.cc:407] Connection header received
0927 00:45:24.581392 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:24.581395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:24.581440 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:24.581515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:24.582562 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.583082 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.584120 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.584334 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.584407 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:24.586928 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:24.586931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:24.586933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:24.586955 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:24.588597 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.589172 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.589176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.589178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.589261 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.589592 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.589596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.589597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.589785 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:24.589895 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:24.590384 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 00:45:24.590479 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:45:24.591710 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.579632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.1:45671)
0927 00:45:24.580037 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:24.580076 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:24.581203 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:24.581660 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:24.581666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:24.581998 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:24.582430 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.582442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.583184 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.583186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:24.584020 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.584028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.584450 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.584452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:24.584576 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:24.585115 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:24.585129 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:24.586787 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:24.588722 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:24.588730 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:24.588746 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:24.589068 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:24.589353 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:24.589357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:24.589359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:24.589490 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:24.589892 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:24.589897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:24.590130 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 00:45:24.590329 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 00:45:24.590666 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260927 00:45:24.592949 19798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260927 00:45:24.593168 19798 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:24.593634 19798 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
I20260927 00:45:24.595000 19607 ts_manager.cc:194] Registered new tserver with Master: e24ffcf91b644d979c5d0f72df302bb8 (127.0.98.1:39317)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 00:45:24.596134 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 00:45:24.611946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:24.718727 19803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:24.718993 19803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:24.719067 19803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:24.723008 19803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:24.723084 19803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:24.723181 19803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:24.728133 19803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:24.729131 19803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:24.730247 19803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:24.739626 19803 server_base.cc:1034] running on GCE node
W20260927 00:45:24.739579 19808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.739588 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.739579 19809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.740090 19803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:24.740690 19803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:24.741846 19803 hybrid_clock.cc:648] HybridClock initialized: now 1790469924741831 us; error 38 us; skew 500 ppm
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:24.744324 19803 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 00:45:24.745200 19803 webserver.cc:460] Webserver started at http://127.0.98.2:41611/ using document root <none> and password file <none>
I20260927 00:45:24.745723 19803 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:24.745806 19803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:24.746028 19803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:24.747905 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "adfedcbf080e4c14887bb52ede0131ca"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3e2858792156dc7611e2bd5575a0493c"
server_key_iv: "756b59ebbf41b1846959c633e7df5c43"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.748468 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "adfedcbf080e4c14887bb52ede0131ca"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3e2858792156dc7611e2bd5575a0493c"
server_key_iv: "756b59ebbf41b1846959c633e7df5c43"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.751901 19803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:24.754328 19818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.755513 19803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:24.755640 19803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "adfedcbf080e4c14887bb52ede0131ca"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "3e2858792156dc7611e2bd5575a0493c"
server_key_iv: "756b59ebbf41b1846959c633e7df5c43"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.755753 19803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:24.777496 19803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:24.780581 19803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:24.780776 19803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:24.781381 19803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:24.782184 19803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:24.782262 19803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.782351 19803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:24.782401 19803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.791627 19803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45697
I20260927 00:45:24.791658 19931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45697 every 8 connection(s)
I20260927 00:45:24.792466 19803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:24.797363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19803
I20260927 00:45:24.797518 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:24.797816 392 external_mini_cluster.cc:1428] Setting key 140272530b7cf65c3bc8977f5f8a6316
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:24.806923 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.794169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60281 (local address 127.0.98.62:44193)
0927 00:45:24.794313 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:24.794318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:24.795188 (+ 870us) server_negotiation.cc:407] Connection header received
0927 00:45:24.796433 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:24.796437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:24.796494 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:24.796570 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:24.797810 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.798541 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.800147 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.800361 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.800440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:24.803071 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:24.803076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:24.803079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:24.803110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:24.804839 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.805356 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.805360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.805362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.805453 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:24.805779 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:24.805782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:24.805784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:24.806019 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:24.806128 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:24.806585 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 00:45:24.806735 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 00:45:24.807675 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.794601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.2:60281)
0927 00:45:24.795037 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:24.795078 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:24.796193 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:24.796728 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:24.796734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:24.797140 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:24.797675 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.797691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.798680 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.798683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:24.800007 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.800034 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.800511 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.800517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:24.800743 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:24.801498 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:24.801514 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:24.802905 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:24.804984 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:24.804989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:24.805000 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:24.805248 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:24.805549 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:24.805550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:24.805552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:24.805675 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:24.806120 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:24.806125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:24.806339 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 00:45:24.806530 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 00:45:24.806844 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260927 00:45:24.808933 19932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260927 00:45:24.809180 19932 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:24.809664 19932 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 00:45:24.810803 19607 ts_manager.cc:194] Registered new tserver with Master: adfedcbf080e4c14887bb52ede0131ca (127.0.98.2:45697)
I20260927 00:45:24.811365 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 00:45:24.825456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:24.932368 19937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:24.932596 19937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:24.932670 19937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:24.936921 19937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:24.936985 19937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:24.937065 19937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:24.942066 19937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:24.942975 19937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:24.943898 19937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:24.952803 19943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.952914 19937 server_base.cc:1034] running on GCE node
W20260927 00:45:24.952813 19945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:24.952804 19942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:24.953518 19937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:24.954183 19937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:24.955343 19937 hybrid_clock.cc:648] HybridClock initialized: now 1790469924955329 us; error 25 us; skew 500 ppm
Sep 27 00:45:24 dist-test-slave-pkws krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:24.958000 19937 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 00:45:24.958861 19937 webserver.cc:460] Webserver started at http://127.0.98.3:41487/ using document root <none> and password file <none>
I20260927 00:45:24.959429 19937 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:24.959517 19937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:24.959736 19937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:24.961687 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec5bc8e153c54eb285aaddb5e25374d6"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "5aea735c98bffea3a2e731d49a5ce1fb"
server_key_iv: "a5a5295a114576d39f73e931bbae7e6a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.962251 19937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec5bc8e153c54eb285aaddb5e25374d6"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "5aea735c98bffea3a2e731d49a5ce1fb"
server_key_iv: "a5a5295a114576d39f73e931bbae7e6a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.965734 19937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:24.967964 19952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.969125 19937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:24.969250 19937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "ec5bc8e153c54eb285aaddb5e25374d6"
format_stamp: "Formatted at 2026-09-27 00:45:24 on dist-test-slave-pkws"
server_key: "5aea735c98bffea3a2e731d49a5ce1fb"
server_key_iv: "a5a5295a114576d39f73e931bbae7e6a"
server_key_version: "encryptionkey@0"
I20260927 00:45:24.969352 19937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:24.979179 19937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:24.981866 19937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:24.982048 19937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:24.982563 19937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:24.983309 19937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:24.983407 19937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.983474 19937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:24.983522 19937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:24.992272 19937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38853
I20260927 00:45:24.992295 20065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38853 every 8 connection(s)
I20260927 00:45:24.993156 19937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:25.001194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 19937
I20260927 00:45:25.001353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:25.001869 392 external_mini_cluster.cc:1428] Setting key 70c05976b295d48988cd1bfeb076cbd1
Sep 27 00:45:25 dist-test-slave-pkws krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:25.006724 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.994802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42573 (local address 127.0.98.62:44193)
0927 00:45:24.994966 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:24.994970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:24.995740 (+ 770us) server_negotiation.cc:407] Connection header received
0927 00:45:24.996971 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:24.996974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:24.997026 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:24.997096 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:24.998200 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.998735 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.999778 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.999986 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.000060 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.002816 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:25.002820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:25.002822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:25.002850 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:25.004775 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:25.005300 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:25.005303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:25.005304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:25.005392 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:25.005726 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:25.005728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:25.005729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:25.005896 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:25.005969 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.006465 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.006601 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:45:25.007557 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:24.995162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.3:42573)
0927 00:45:24.995599 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:24.995639 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:24.996743 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:24.997248 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:24.997255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:24.997604 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:24.998061 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.998074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:24.998837 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:24.998840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:24.999675 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:24.999683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.000107 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.000109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.000242 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.000732 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:25.000746 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:25.002628 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:25.004933 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:25.004942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:25.004955 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:25.005215 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:25.005496 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:25.005499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:25.005501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:25.005646 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:25.005981 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:25.005986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:25.006265 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.006415 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.006697 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":115,"threads_started":1}
I20260927 00:45:25.008584 20066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260927 00:45:25.008808 20066 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:25.009284 20066 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
I20260927 00:45:25.010219 19607 ts_manager.cc:194] Registered new tserver with Master: ec5bc8e153c54eb285aaddb5e25374d6 (127.0.98.3:38853)
I20260927 00:45:25.010818 19607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42573
I20260927 00:45:25.015609 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:25.024400 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.017508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.98.62:44193)
0927 00:45:25.017648 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.017653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.017769 (+ 116us) server_negotiation.cc:407] Connection header received
0927 00:45:25.017858 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.017860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.017897 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.017972 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:25.018634 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.019131 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.019913 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.020119 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.020214 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.021523 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:25.021527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:25.021528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:25.021548 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:25.022765 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:25.023170 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:25.023173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:25.023175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:25.023258 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:25.023506 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:25.023509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:25.023510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:25.023732 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:25.023846 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.024058 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.024218 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 00:45:25.027117 19607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:25.030097 19607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:25.043561 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.038216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.98.2:45697)
0927 00:45:25.038578 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.038590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.038614 (+ 24us) server_negotiation.cc:407] Connection header received
0927 00:45:25.038812 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.038818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.039052 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.039203 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:25.039692 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.040799 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.042101 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.042909 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.043010 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.043150 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.043321 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.043412 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":165,"threads_started":1}
I20260927 00:45:25.043551 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.038076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45697 (local address 127.0.0.1:39138)
0927 00:45:25.038466 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.038515 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.038642 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.039400 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.039406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.039430 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:25.039567 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.039572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.040931 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.040936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.041959 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.041968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.043023 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.043027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.043127 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.043162 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.043293 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.043408 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260927 00:45:25.044201 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.038810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38853 (local address 127.0.0.1:47004)
0927 00:45:25.039455 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.039467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.039563 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.040189 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.040192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.040204 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:25.040289 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.040294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.041664 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.041668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.042807 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.042817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.043834 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.043840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.043942 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.043965 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.044009 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.044068 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":98,"threads_started":1}
I20260927 00:45:25.044643 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.039487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.98.1:39317)
0927 00:45:25.039834 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.039845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.040093 (+ 248us) server_negotiation.cc:407] Connection header received
0927 00:45:25.040175 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.040179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.040343 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.040486 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:25.040661 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.041737 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.043012 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.043877 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.043943 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.044069 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.044409 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.044500 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260927 00:45:25.044780 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.038999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.98.3:38853)
0927 00:45:25.039448 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.039460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.039539 (+ 79us) server_negotiation.cc:407] Connection header received
0927 00:45:25.039668 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.039673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.040064 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.040209 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:25.040378 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.041521 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.042940 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.043701 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.044379 (+ 678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.044505 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.044546 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.044620 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":137,"threads_started":1}
I20260927 00:45:25.045058 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.039088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:33100)
0927 00:45:25.039974 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.039986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.040079 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.040476 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.040479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.040495 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:25.040573 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.040578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.041860 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.041866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.042843 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.042852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.044154 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.044157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.044242 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.044266 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.044821 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.044918 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260927 00:45:25.046929 19866 tablet_service.cc:1467] Processing CreateTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc (DEFAULT_TABLE table=test-table [id=075c788de172447fbd5f270a3b5ad220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:25.047097 20000 tablet_service.cc:1467] Processing CreateTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc (DEFAULT_TABLE table=test-table [id=075c788de172447fbd5f270a3b5ad220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:25.047964 20000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47aaa3c4b6904b4d96cae13420f82cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:25.047979 19866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47aaa3c4b6904b4d96cae13420f82cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:25.051257 19732 tablet_service.cc:1467] Processing CreateTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc (DEFAULT_TABLE table=test-table [id=075c788de172447fbd5f270a3b5ad220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:25.052222 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47aaa3c4b6904b4d96cae13420f82cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:25.054503 20083 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Bootstrap starting.
I20260927 00:45:25.054975 20084 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Bootstrap starting.
I20260927 00:45:25.056631 20083 tablet_bootstrap.cc:654] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:25.057120 20084 tablet_bootstrap.cc:654] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:25.057886 20083 log.cc:824] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:25.058336 20084 log.cc:824] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:25.059406 20086 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Bootstrap starting.
I20260927 00:45:25.059944 20083 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: No bootstrap required, opened a new log
I20260927 00:45:25.060163 20083 ts_tablet_manager.cc:1397] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:45:25.060258 20084 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: No bootstrap required, opened a new log
I20260927 00:45:25.060432 20084 ts_tablet_manager.cc:1397] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 00:45:25.061435 20086 tablet_bootstrap.cc:654] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:25.062552 20086 log.cc:824] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:25.063596 20083 raft_consensus.cc:348] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.063926 20083 raft_consensus.cc:374] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:25.064002 20083 raft_consensus.cc:729] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adfedcbf080e4c14887bb52ede0131ca, State: Initialized, Role: FOLLOWER
I20260927 00:45:25.064361 20086 tablet_bootstrap.cc:492] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: No bootstrap required, opened a new log
I20260927 00:45:25.064302 20084 raft_consensus.cc:348] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.064531 20086 ts_tablet_manager.cc:1397] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:25.064452 20083 consensus_queue.cc:260] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.064574 20084 raft_consensus.cc:374] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:25.064637 20084 raft_consensus.cc:729] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5bc8e153c54eb285aaddb5e25374d6, State: Initialized, Role: FOLLOWER
I20260927 00:45:25.065094 20084 consensus_queue.cc:260] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.065177 19932 heartbeater.cc:503] Master 127.0.98.62:44193 was elected leader, sending a full tablet report...
I20260927 00:45:25.065580 20083 ts_tablet_manager.cc:1428] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 00:45:25.065805 20066 heartbeater.cc:503] Master 127.0.98.62:44193 was elected leader, sending a full tablet report...
I20260927 00:45:25.066030 20084 ts_tablet_manager.cc:1428] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:45:25.070508 20086 raft_consensus.cc:348] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.070719 20086 raft_consensus.cc:374] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:25.070775 20086 raft_consensus.cc:729] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24ffcf91b644d979c5d0f72df302bb8, State: Initialized, Role: FOLLOWER
I20260927 00:45:25.071118 20086 consensus_queue.cc:260] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.071687 19798 heartbeater.cc:503] Master 127.0.98.62:44193 was elected leader, sending a full tablet report...
I20260927 00:45:25.071954 20086 ts_tablet_manager.cc:1428] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260927 00:45:25.078691 19799 tablet.cc:2369] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 00:45:25.244020 20067 tablet.cc:2369] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:25.261925 20090 raft_consensus.cc:482] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:25.262148 20090 raft_consensus.cc:504] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.263370 20090 leader_election.cc:288] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adfedcbf080e4c14887bb52ede0131ca (127.0.98.2:45697), e24ffcf91b644d979c5d0f72df302bb8 (127.0.98.1:39317)
I20260927 00:45:25.267139 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.263639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.98.3:51883)
0927 00:45:25.263863 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.263880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.263981 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.264317 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.264320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.264337 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:25.264451 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.264457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.265266 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.265270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.266098 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.266107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.266718 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.266827 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.266851 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.266939 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.267008 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":21}
I20260927 00:45:25.267140 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.263765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51883 (local address 127.0.98.1:39317)
0927 00:45:25.263903 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.263908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.263929 (+ 21us) server_negotiation.cc:407] Connection header received
0927 00:45:25.264147 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.264150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.264220 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.264298 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:25.264576 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.265152 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266221 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.266603 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266718 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.266781 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.266954 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.267029 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 00:45:25.267325 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.263833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40083 (local address 127.0.98.2:45697)
0927 00:45:25.263951 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.263955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.264206 (+ 251us) server_negotiation.cc:407] Connection header received
0927 00:45:25.264379 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.264382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.264428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.264519 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:25.264773 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.265329 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266396 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.266795 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266920 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.266980 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.267152 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.267217 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 00:45:25.267354 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.263646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45697 (local address 127.0.98.3:40083)
0927 00:45:25.264102 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.264116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.264256 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.264527 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.264529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.264544 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:25.264665 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.264670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.265439 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.265444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.266259 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.266268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.266920 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.266924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.267023 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.267043 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.267137 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.267200 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":57,"thread_start_us":143,"threads_started":1}
I20260927 00:45:25.267871 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47aaa3c4b6904b4d96cae13420f82cbc" candidate_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24ffcf91b644d979c5d0f72df302bb8" is_pre_election: true
I20260927 00:45:25.268097 19886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47aaa3c4b6904b4d96cae13420f82cbc" candidate_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfedcbf080e4c14887bb52ede0131ca" is_pre_election: true
I20260927 00:45:25.268162 19752 raft_consensus.cc:2454] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5bc8e153c54eb285aaddb5e25374d6 in term 0.
I20260927 00:45:25.268401 19886 raft_consensus.cc:2454] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5bc8e153c54eb285aaddb5e25374d6 in term 0.
I20260927 00:45:25.268649 19956 leader_election.cc:302] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e24ffcf91b644d979c5d0f72df302bb8, ec5bc8e153c54eb285aaddb5e25374d6; no voters:
I20260927 00:45:25.268929 20090 raft_consensus.cc:2781] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:25.269001 20090 raft_consensus.cc:482] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:25.269042 20090 raft_consensus.cc:3037] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:25.270097 20090 raft_consensus.cc:504] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.270457 20090 leader_election.cc:288] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [CANDIDATE]: Term 1 election: Requested vote from peers adfedcbf080e4c14887bb52ede0131ca (127.0.98.2:45697), e24ffcf91b644d979c5d0f72df302bb8 (127.0.98.1:39317)
I20260927 00:45:25.270783 19886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47aaa3c4b6904b4d96cae13420f82cbc" candidate_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adfedcbf080e4c14887bb52ede0131ca"
I20260927 00:45:25.270820 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47aaa3c4b6904b4d96cae13420f82cbc" candidate_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24ffcf91b644d979c5d0f72df302bb8"
I20260927 00:45:25.270882 19886 raft_consensus.cc:3037] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:25.270967 19752 raft_consensus.cc:3037] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:25.271931 19886 raft_consensus.cc:2454] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5bc8e153c54eb285aaddb5e25374d6 in term 1.
I20260927 00:45:25.271981 19752 raft_consensus.cc:2454] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5bc8e153c54eb285aaddb5e25374d6 in term 1.
I20260927 00:45:25.272245 19955 leader_election.cc:302] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adfedcbf080e4c14887bb52ede0131ca, ec5bc8e153c54eb285aaddb5e25374d6; no voters:
I20260927 00:45:25.272409 20090 raft_consensus.cc:2781] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:25.272605 20090 raft_consensus.cc:686] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 1 LEADER]: Becoming Leader. State: Replica: ec5bc8e153c54eb285aaddb5e25374d6, State: Running, Role: LEADER
I20260927 00:45:25.272979 20090 consensus_queue.cc:237] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } }
I20260927 00:45:25.275892 19607 catalog_manager.cc:5495] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5bc8e153c54eb285aaddb5e25374d6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5bc8e153c54eb285aaddb5e25374d6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 } health_report { overall_health: UNKNOWN } } }
W20260927 00:45:25.293664 19933 tablet.cc:2369] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:25.333997 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.330452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.0.98.3:38853)
0927 00:45:25.330592 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.330597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.330613 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:45:25.330722 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.330724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.330778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.330899 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:25.331142 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.331645 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.332501 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.332726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.332834 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.333200 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:25.333598 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:25.333732 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.333779 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.333834 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 00:45:25.339869 19752 raft_consensus.cc:1261] T 47aaa3c4b6904b4d96cae13420f82cbc P e24ffcf91b644d979c5d0f72df302bb8 [term 1 FOLLOWER]: Refusing update from remote peer ec5bc8e153c54eb285aaddb5e25374d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:25.339869 19886 raft_consensus.cc:1261] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 1 FOLLOWER]: Refusing update from remote peer ec5bc8e153c54eb285aaddb5e25374d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:25.340471 20093 consensus_queue.cc:1035] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24ffcf91b644d979c5d0f72df302bb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:25.340649 20090 consensus_queue.cc:1035] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adfedcbf080e4c14887bb52ede0131ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:25.347913 20097 mvcc.cc:205] Tried to move back new op lower bound from 7333764814185279488 to 7333764813920931840. Current Snapshot: MvccSnapshot[applied={T|T < 7333764814185279488}]
I20260927 00:45:25.348001 20098 mvcc.cc:205] Tried to move back new op lower bound from 7333764814185279488 to 7333764813920931840. Current Snapshot: MvccSnapshot[applied={T|T < 7333764814185279488}]
I20260927 00:45:25.355629 19607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:25.355784 19607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:25.358084 19607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40feddb2077d4a09a412358f8240f144: Sending DeleteTablet for 3 replicas of tablet 47aaa3c4b6904b4d96cae13420f82cbc
I20260927 00:45:25.358783 19866 tablet_service.cc:1514] Processing DeleteTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39138
I20260927 00:45:25.358914 20000 tablet_service.cc:1514] Processing DeleteTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47004
I20260927 00:45:25.359263 20106 tablet_replica.cc:330] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: stopping tablet replica
I20260927 00:45:25.359380 20107 tablet_replica.cc:330] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: stopping tablet replica
I20260927 00:45:25.359632 20106 raft_consensus.cc:2229] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:25.359691 20107 raft_consensus.cc:2229] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:25.359731 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 47aaa3c4b6904b4d96cae13420f82cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33100
I20260927 00:45:25.360002 20106 raft_consensus.cc:2258] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:25.360098 20107 raft_consensus.cc:2258] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:25.360525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19669
I20260927 00:45:25.361492 20107 ts_tablet_manager.cc:1905] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:25.361485 20106 ts_tablet_manager.cc:1905] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:25.364113 20106 ts_tablet_manager.cc:1918] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:25.364193 20106 log.cc:1196] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/47aaa3c4b6904b4d96cae13420f82cbc
I20260927 00:45:25.364494 20106 ts_tablet_manager.cc:1939] T 47aaa3c4b6904b4d96cae13420f82cbc P adfedcbf080e4c14887bb52ede0131ca: Deleting consensus metadata
I20260927 00:45:25.364745 20107 ts_tablet_manager.cc:1918] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:25.364826 20107 log.cc:1196] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/47aaa3c4b6904b4d96cae13420f82cbc
I20260927 00:45:25.365175 20107 ts_tablet_manager.cc:1939] T 47aaa3c4b6904b4d96cae13420f82cbc P ec5bc8e153c54eb285aaddb5e25374d6: Deleting consensus metadata
I20260927 00:45:25.365635 19593 catalog_manager.cc:4841] TS adfedcbf080e4c14887bb52ede0131ca (127.0.98.2:45697): tablet 47aaa3c4b6904b4d96cae13420f82cbc (table test-table [id=075c788de172447fbd5f270a3b5ad220]) successfully deleted
I20260927 00:45:25.366216 19593 catalog_manager.cc:4841] TS ec5bc8e153c54eb285aaddb5e25374d6 (127.0.98.3:38853): tablet 47aaa3c4b6904b4d96cae13420f82cbc (table test-table [id=075c788de172447fbd5f270a3b5ad220]) successfully deleted
W20260927 00:45:25.369853 19594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39317 (error 108)
I20260927 00:45:25.370635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19803
I20260927 00:45:25.370908 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.370577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39317 (local address 127.0.0.1:33102)
0927 00:45:25.370727 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.370804 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 00:45:25.371268 19594 catalog_manager.cc:4584] TS e24ffcf91b644d979c5d0f72df302bb8 (127.0.98.1:39317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47aaa3c4b6904b4d96cae13420f82cbc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39317: connect: Connection refused (error 111)
I20260927 00:45:25.377205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19937
I20260927 00:45:25.382031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 19575
2026-09-27T00:45:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2426 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T00:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:25Z Disabled control of system clock
I20260927 00:45:25.411561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:25.519505 20113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:25.519749 20113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:25.519783 20113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:25.523609 20113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:25.523658 20113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:25.523674 20113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:25.523690 20113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:25.523705 20113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:25.528427 20113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:25.529376 20113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:25.530623 20113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:25.538775 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:25.538882 20113 server_base.cc:1034] running on GCE node
W20260927 00:45:25.538826 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:25.538810 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:25.539590 20113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:25.540738 20113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:25.541914 20113 hybrid_clock.cc:648] HybridClock initialized: now 1790469925541884 us; error 48 us; skew 500 ppm
I20260927 00:45:25.543730 20113 webserver.cc:460] Webserver started at http://127.0.98.62:45703/ using document root <none> and password file <none>
I20260927 00:45:25.544335 20113 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:25.544400 20113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:25.544654 20113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:25.546797 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "2d9dfbd99b4049dc891616bab5d7b495"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "f834599e13884e572a083ac882faa414"
server_key_iv: "63d723e26b5b3ab21c87c586051ffd9c"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.547405 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d9dfbd99b4049dc891616bab5d7b495"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "f834599e13884e572a083ac882faa414"
server_key_iv: "63d723e26b5b3ab21c87c586051ffd9c"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.551144 20113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:25.553512 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:25.554668 20113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:25.554813 20113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "2d9dfbd99b4049dc891616bab5d7b495"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "f834599e13884e572a083ac882faa414"
server_key_iv: "63d723e26b5b3ab21c87c586051ffd9c"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.554922 20113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:25.577091 20113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:25.577718 20113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:25.577910 20113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:25.584401 20113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34049
I20260927 00:45:25.584414 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34049 every 8 connection(s)
I20260927 00:45:25.585325 20113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:25.586704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20113
I20260927 00:45:25.586871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:25.587175 392 external_mini_cluster.cc:1428] Setting key d21e73b439a2647d002210e2a8d08e3e
I20260927 00:45:25.588735 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:25.595402 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: Bootstrap starting.
I20260927 00:45:25.597538 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.588427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.0.98.62:34049)
0927 00:45:25.589088 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.589106 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.589146 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:45:25.590030 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.590051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.590511 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.590851 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:25.591449 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.592882 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.593784 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.594516 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.594673 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.595493 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:25.595499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:25.595515 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:25.595971 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:25.596067 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:25.596093 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:25.596187 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.596334 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.596626 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260927 00:45:25.598024 20180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:25.598835 20180 log.cc:824] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:25.600652 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: No bootstrap required, opened a new log
I20260927 00:45:25.603967 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260927 00:45:25.604211 20180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:25.604250 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9dfbd99b4049dc891616bab5d7b495, State: Initialized, Role: FOLLOWER
I20260927 00:45:25.604614 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260927 00:45:25.604719 20180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:25.604764 20180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:25.604830 20180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:25.605979 20180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260927 00:45:25.606274 20180 leader_election.cc:302] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9dfbd99b4049dc891616bab5d7b495; no voters:
I20260927 00:45:25.606518 20180 leader_election.cc:288] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:25.606632 20185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:25.606861 20185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [term 1 LEADER]: Becoming Leader. State: Replica: 2d9dfbd99b4049dc891616bab5d7b495, State: Running, Role: LEADER
I20260927 00:45:25.607211 20185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } }
I20260927 00:45:25.607647 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:25.608771 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9dfbd99b4049dc891616bab5d7b495. Latest consensus state: current_term: 1 leader_uuid: "2d9dfbd99b4049dc891616bab5d7b495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } }
I20260927 00:45:25.608922 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:25.609247 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d9dfbd99b4049dc891616bab5d7b495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9dfbd99b4049dc891616bab5d7b495" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34049 } } }
I20260927 00:45:25.609352 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:25.609330 20194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:25.612102 20194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:25.617070 20194 catalog_manager.cc:1347] Generated new cluster ID: bc01c0c9a4d1483792a81e82eb70caa2
I20260927 00:45:25.617141 20194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:25.667430 20194 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:25.668380 20194 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:25.682837 20194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: Generated new TSK 0
I20260927 00:45:25.683531 20194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:25.689415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 00:45:25.796226 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:25.796481 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:25.796517 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:25.800419 20204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:25.800472 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:25.800523 20204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 00:45:25.805356 20204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:25.806348 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:25.807652 20204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:25.816962 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:25.816972 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:25.817157 20204 server_base.cc:1034] running on GCE node
W20260927 00:45:25.817332 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:25.817735 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:25.818341 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:25.819521 20204 hybrid_clock.cc:648] HybridClock initialized: now 1790469925819504 us; error 36 us; skew 500 ppm
I20260927 00:45:25.821274 20204 webserver.cc:460] Webserver started at http://127.0.98.1:44343/ using document root <none> and password file <none>
I20260927 00:45:25.821825 20204 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:25.821949 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:25.822165 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:25.824172 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data/instance:
uuid: "f886d80c20e14ab7a132b22a4e5ae440"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "623a8c40611494fae01b4243e88f24c8"
server_key_iv: "001b9f610df9e8671cbe87f5025b9774"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.824990 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f886d80c20e14ab7a132b22a4e5ae440"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "623a8c40611494fae01b4243e88f24c8"
server_key_iv: "001b9f610df9e8671cbe87f5025b9774"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.830133 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 00:45:25.832432 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:25.833597 20204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:25.833726 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal
uuid: "f886d80c20e14ab7a132b22a4e5ae440"
format_stamp: "Formatted at 2026-09-27 00:45:25 on dist-test-slave-pkws"
server_key: "623a8c40611494fae01b4243e88f24c8"
server_key_iv: "001b9f610df9e8671cbe87f5025b9774"
server_key_version: "encryptionkey@0"
I20260927 00:45:25.833835 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:25.854171 20204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:25.854813 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:25.855006 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:25.855564 20204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:25.856480 20204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:25.856552 20204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:25.856621 20204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:25.856663 20204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:25.865415 20204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44145
I20260927 00:45:25.865442 20331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44145 every 8 connection(s)
I20260927 00:45:25.866293 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/data/info.pb
I20260927 00:45:25.874868 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.868075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44311 (local address 127.0.98.62:34049)
0927 00:45:25.868219 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:25.868223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:25.869226 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 00:45:25.870559 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:25.870561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:25.870610 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:25.870684 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:25.871538 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.872054 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.873221 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.873428 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.873508 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.874452 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:25.874456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:25.874458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:25.874469 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:25.874487 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:25.874495 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:25.874550 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:25.874675 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 00:45:25.874757 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:45:25.875592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20204
I20260927 00:45:25.875654 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:25.868436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.1:44311)
0927 00:45:25.869106 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:25.869145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:25.870366 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:25.870840 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:25.870847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:25.870932 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:25.871419 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.871432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.872169 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.872171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.873123 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:25.873132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:25.873558 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:25.873561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:25.873692 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:25.874216 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:25.874218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:25.874253 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:25.874308 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:25.874564 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:25.874570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:25.874574 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:45:25.874659 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 00:45:25.874872 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260927 00:45:25.875754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-0/wal/instance
I20260927 00:45:25.876107 392 external_mini_cluster.cc:1428] Setting key 4810a66a4b3ebed0ca316869c2a50ee2
I20260927 00:45:25.876976 20332 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260927 00:45:25.877210 20332 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:25.877700 20332 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260927 00:45:25.878029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:25.879232 20144 ts_manager.cc:194] Registered new tserver with Master: f886d80c20e14ab7a132b22a4e5ae440 (127.0.98.1:44145)
I20260927 00:45:25.880347 20144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44311
W20260927 00:45:25.984530 20335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:25.984805 20335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:25.984877 20335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:25.988797 20335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:25.988874 20335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:25.988976 20335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 00:45:25.993642 20335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:25.994632 20335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:25.995891 20335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:26.005044 20343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:26.005084 20340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:26.005084 20341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:26.005218 20335 server_base.cc:1034] running on GCE node
I20260927 00:45:26.005692 20335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:26.006312 20335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:26.007476 20335 hybrid_clock.cc:648] HybridClock initialized: now 1790469926007459 us; error 40 us; skew 500 ppm
I20260927 00:45:26.009346 20335 webserver.cc:460] Webserver started at http://127.0.98.2:41163/ using document root <none> and password file <none>
I20260927 00:45:26.009948 20335 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:26.010008 20335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:26.010210 20335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:26.012169 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data/instance:
uuid: "4568cc8cd71f46aeb5c27994ff4f1401"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "fd1a7ca8c15373490dd391e427e4de3e"
server_key_iv: "b190a63f39117f377c79e5f3024f9e44"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.012715 20335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4568cc8cd71f46aeb5c27994ff4f1401"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "fd1a7ca8c15373490dd391e427e4de3e"
server_key_iv: "b190a63f39117f377c79e5f3024f9e44"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.016189 20335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:45:26.018496 20349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.019667 20335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:26.019797 20335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal
uuid: "4568cc8cd71f46aeb5c27994ff4f1401"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "fd1a7ca8c15373490dd391e427e4de3e"
server_key_iv: "b190a63f39117f377c79e5f3024f9e44"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.019927 20335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:26.032847 20335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:26.033525 20335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:26.033706 20335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:26.034232 20335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:26.035101 20335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:26.035171 20335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.035234 20335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:26.035270 20335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.043780 20335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33637
I20260927 00:45:26.043821 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33637 every 8 connection(s)
I20260927 00:45:26.044670 20335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/data/info.pb
I20260927 00:45:26.053534 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.046414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59275 (local address 127.0.98.62:34049)
0927 00:45:26.046558 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.046561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.047526 (+ 965us) server_negotiation.cc:407] Connection header received
0927 00:45:26.048933 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.048937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.049003 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.049106 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:26.049960 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.050504 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.051632 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.051859 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.051978 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.053047 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:26.053051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:26.053053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:26.053064 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:26.053082 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:26.053090 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:26.053169 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.053324 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.053389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:45:26.054183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20335
I20260927 00:45:26.054332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal/instance
I20260927 00:45:26.054361 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.046776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.2:59275)
0927 00:45:26.047388 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.047426 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.048676 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.049277 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.049284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.049362 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:26.049823 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.049836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.050650 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.050654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.051508 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.051517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.052015 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.052020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.052177 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.052729 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:26.052730 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:26.052767 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:26.052824 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:26.053170 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:26.053182 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:26.053190 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.053323 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.053608 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260927 00:45:26.054716 392 external_mini_cluster.cc:1428] Setting key d7305682eb79596327f9bbce0dcef414
I20260927 00:45:26.055632 20463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260927 00:45:26.055855 20463 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:26.056403 20463 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260927 00:45:26.056756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--builtin_ntp_servers=127.0.98.20:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 00:45:26.057448 20144 ts_manager.cc:194] Registered new tserver with Master: 4568cc8cd71f46aeb5c27994ff4f1401 (127.0.98.2:33637)
I20260927 00:45:26.057998 20144 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59275
W20260927 00:45:26.165162 20466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:26.165388 20466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:26.165424 20466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:26.169360 20466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:26.169407 20466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:26.169483 20466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 00:45:26.174197 20466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:26.175179 20466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:26.176481 20466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:26.185667 20472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:26.185690 20471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:26.185699 20474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:26.185746 20466 server_base.cc:1034] running on GCE node
I20260927 00:45:26.186261 20466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:26.186832 20466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:26.187990 20466 hybrid_clock.cc:648] HybridClock initialized: now 1790469926187978 us; error 31 us; skew 500 ppm
I20260927 00:45:26.189704 20466 webserver.cc:460] Webserver started at http://127.0.98.3:36443/ using document root <none> and password file <none>
I20260927 00:45:26.190222 20466 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:26.190279 20466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:26.190496 20466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:26.192404 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e13b7be681b4041abd625360f21b4e0"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "b76383652edb8fcf06d0969f6aef0221"
server_key_iv: "5037298445c7a76a7461eaedeefe1b39"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.193001 20466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e13b7be681b4041abd625360f21b4e0"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "b76383652edb8fcf06d0969f6aef0221"
server_key_iv: "5037298445c7a76a7461eaedeefe1b39"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.196310 20466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:45:26.198580 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.199648 20466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 00:45:26.199757 20466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal
uuid: "3e13b7be681b4041abd625360f21b4e0"
format_stamp: "Formatted at 2026-09-27 00:45:26 on dist-test-slave-pkws"
server_key: "b76383652edb8fcf06d0969f6aef0221"
server_key_iv: "5037298445c7a76a7461eaedeefe1b39"
server_key_version: "encryptionkey@0"
I20260927 00:45:26.199860 20466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:26.221257 20466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:26.221995 20466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:26.222179 20466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:26.222764 20466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 00:45:26.223737 20466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 00:45:26.223809 20466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.223886 20466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 00:45:26.223917 20466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:26.232162 20466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44333
I20260927 00:45:26.232187 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44333 every 8 connection(s)
I20260927 00:45:26.233000 20466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/data/info.pb
I20260927 00:45:26.241550 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.234677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58401 (local address 127.0.98.62:34049)
0927 00:45:26.234805 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.234809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.235815 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 00:45:26.237253 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.237256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.237300 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.237379 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:26.238287 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.238820 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.239873 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.240090 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.240166 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.241175 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:26.241178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:26.241180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:26.241191 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:26.241208 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:26.241215 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:26.241270 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.241400 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.241445 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 00:45:26.242291 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.235047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34049 (local address 127.0.98.3:58401)
0927 00:45:26.235645 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.235701 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.237056 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.237552 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.237559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.237635 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:26.238134 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.238147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.238926 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.238929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.239768 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.239776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.240214 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.240217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.240356 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.240877 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 00:45:26.240878 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:26.240973 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 00:45:26.241047 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:26.241286 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:26.241292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:26.241296 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.241380 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.241572 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
I20260927 00:45:26.242743 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20466
I20260927 00:45:26.242872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal/instance
I20260927 00:45:26.243147 392 external_mini_cluster.cc:1428] Setting key 9d49a94f04f1a5e52cfabcb540c5280b
I20260927 00:45:26.243361 20594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34049
I20260927 00:45:26.243571 20594 heartbeater.cc:464] Registering TS with master...
I20260927 00:45:26.244128 20594 heartbeater.cc:511] Master 127.0.98.62:34049 requested a full tablet report, sending...
I20260927 00:45:26.245286 20143 ts_manager.cc:194] Registered new tserver with Master: 3e13b7be681b4041abd625360f21b4e0 (127.0.98.3:44333)
I20260927 00:45:26.245836 20143 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58401
I20260927 00:45:26.256750 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 00:45:26.262348 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.258627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.0.98.62:34049)
0927 00:45:26.258767 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.258772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.258916 (+ 144us) server_negotiation.cc:407] Connection header received
0927 00:45:26.259063 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.259067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.259114 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.259197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:26.259473 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.259959 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.260831 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.261126 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.261237 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.261948 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:26.261952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:26.261955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:26.261967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:26.261990 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:26.262004 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:26.262079 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.262167 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.262215 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 00:45:26.264881 20143 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 00:45:26.267747 20143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 00:45:26.281294 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.276022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44145 (local address 127.0.0.1:48486)
0927 00:45:26.276302 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.276315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.276402 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.277268 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.277271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.277303 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:26.277478 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.277485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.278893 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.278896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.279956 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.279963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.280750 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.280753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.280863 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.280921 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.281023 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.281137 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":53,"spinlock_wait_cycles":8960,"thread_start_us":83,"threads_started":1}
I20260927 00:45:26.281296 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.276379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.98.1:44145)
0927 00:45:26.276772 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.276784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.276807 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:45:26.276923 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.276928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.277110 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.277365 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:26.277577 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.278721 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.280064 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.280623 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.280751 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.280977 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.281048 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.281140 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260927 00:45:26.281461 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.275363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33637 (local address 127.0.0.1:44946)
0927 00:45:26.275754 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.275803 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.275926 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.277138 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.277144 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.277166 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:26.277299 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.277305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.278355 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.278361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.279694 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.279705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.281042 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.281046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.281134 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.281156 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.281233 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.281305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":48,"thread_start_us":62,"threads_started":1}
I20260927 00:45:26.281639 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.275705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44333 (local address 127.0.0.1:45812)
0927 00:45:26.276044 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.276059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.276143 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.277268 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.277271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.277298 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:26.277486 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.277493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.279233 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.279235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.280264 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.280274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.281283 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.281286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.281387 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.281406 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.281453 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.281523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":99,"threads_started":1}
I20260927 00:45:26.281908 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.275457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.98.2:33637)
0927 00:45:26.276230 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.276247 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.276651 (+ 404us) server_negotiation.cc:407] Connection header received
0927 00:45:26.276749 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.276753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.276982 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.277151 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:26.277449 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.278237 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.279839 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.280674 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.280771 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.280975 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.281614 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.281715 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":124,"threads_started":1}
I20260927 00:45:26.282630 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.276427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.3:44333)
0927 00:45:26.276773 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.276784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.276807 (+ 23us) server_negotiation.cc:407] Connection header received
0927 00:45:26.276880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.276924 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.277110 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.277638 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:26.277767 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.279111 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.280423 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.281158 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.282115 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.282288 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.282344 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.282437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":152,"threads_started":1}
I20260927 00:45:26.284855 20528 tablet_service.cc:1467] Processing CreateTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 (DEFAULT_TABLE table=test-table [id=06c50c6fd17644c1b69bb408b5fecbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:26.285110 20266 tablet_service.cc:1467] Processing CreateTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 (DEFAULT_TABLE table=test-table [id=06c50c6fd17644c1b69bb408b5fecbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:26.285104 20397 tablet_service.cc:1467] Processing CreateTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 (DEFAULT_TABLE table=test-table [id=06c50c6fd17644c1b69bb408b5fecbfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 00:45:26.285912 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26dfc37b6a04361bc3c0b6dbe687a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:26.286062 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26dfc37b6a04361bc3c0b6dbe687a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:26.286062 20397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b26dfc37b6a04361bc3c0b6dbe687a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:26.291508 20611 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Bootstrap starting.
I20260927 00:45:26.292845 20612 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: Bootstrap starting.
I20260927 00:45:26.294011 20611 tablet_bootstrap.cc:654] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:26.294870 20612 tablet_bootstrap.cc:654] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:26.294842 20611 log.cc:824] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:26.295567 20613 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Bootstrap starting.
I20260927 00:45:26.295703 20612 log.cc:824] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:26.297384 20611 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: No bootstrap required, opened a new log
I20260927 00:45:26.297569 20613 tablet_bootstrap.cc:654] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:26.297573 20611 ts_tablet_manager.cc:1397] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 00:45:26.297607 20612 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: No bootstrap required, opened a new log
I20260927 00:45:26.297809 20612 ts_tablet_manager.cc:1397] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 00:45:26.298381 20613 log.cc:824] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:26.300246 20613 tablet_bootstrap.cc:492] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: No bootstrap required, opened a new log
I20260927 00:45:26.300426 20613 ts_tablet_manager.cc:1397] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 00:45:26.301314 20611 raft_consensus.cc:348] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.301549 20611 raft_consensus.cc:374] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:26.301599 20611 raft_consensus.cc:729] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e13b7be681b4041abd625360f21b4e0, State: Initialized, Role: FOLLOWER
I20260927 00:45:26.302021 20611 consensus_queue.cc:260] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.302006 20612 raft_consensus.cc:348] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.302264 20612 raft_consensus.cc:374] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:26.302328 20612 raft_consensus.cc:729] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f886d80c20e14ab7a132b22a4e5ae440, State: Initialized, Role: FOLLOWER
I20260927 00:45:26.302721 20594 heartbeater.cc:503] Master 127.0.98.62:34049 was elected leader, sending a full tablet report...
I20260927 00:45:26.302738 20612 consensus_queue.cc:260] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.303002 20611 ts_tablet_manager.cc:1428] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 00:45:26.303504 20332 heartbeater.cc:503] Master 127.0.98.62:34049 was elected leader, sending a full tablet report...
I20260927 00:45:26.303666 20612 ts_tablet_manager.cc:1428] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 00:45:26.305383 20613 raft_consensus.cc:348] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.305578 20613 raft_consensus.cc:374] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:26.305620 20613 raft_consensus.cc:729] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4568cc8cd71f46aeb5c27994ff4f1401, State: Initialized, Role: FOLLOWER
I20260927 00:45:26.306020 20613 consensus_queue.cc:260] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.307163 20463 heartbeater.cc:503] Master 127.0.98.62:34049 was elected leader, sending a full tablet report...
I20260927 00:45:26.307464 20613 ts_tablet_manager.cc:1428] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 00:45:26.310209 20619 raft_consensus.cc:482] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 00:45:26.310355 20619 raft_consensus.cc:504] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.311220 20619 leader_election.cc:288] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f886d80c20e14ab7a132b22a4e5ae440 (127.0.98.1:44145), 3e13b7be681b4041abd625360f21b4e0 (127.0.98.3:44333)
I20260927 00:45:26.314819 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.311414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44333 (local address 127.0.98.2:45415)
0927 00:45:26.311509 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.311525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.311602 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.311936 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.311938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.311954 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:26.312075 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.312084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.312800 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.312805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.313666 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.313675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.314374 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.314378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.314474 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.314522 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.314599 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.314688 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260927 00:45:26.314831 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.311631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45415 (local address 127.0.98.3:44333)
0927 00:45:26.311740 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.311750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.311760 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:45:26.311794 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.311796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.311833 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.311911 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:26.312207 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.312690 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.313786 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.314259 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.314372 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.314426 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.314647 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.314722 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:45:26.315363 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26dfc37b6a04361bc3c0b6dbe687a13" candidate_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e13b7be681b4041abd625360f21b4e0" is_pre_election: true
I20260927 00:45:26.315536 20548 raft_consensus.cc:2454] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4568cc8cd71f46aeb5c27994ff4f1401 in term 0.
I20260927 00:45:26.315937 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.311414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44145 (local address 127.0.98.2:42843)
0927 00:45:26.312271 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.312283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:26.312396 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:26.312656 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:26.312658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:26.312673 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 00:45:26.312791 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.312796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.313551 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.313556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.314532 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:26.314541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.315546 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.315548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:26.315622 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.315669 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 00:45:26.315736 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 00:45:26.315800 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":588,"thread_start_us":101,"threads_started":1}
I20260927 00:45:26.316006 20350 leader_election.cc:302] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e13b7be681b4041abd625360f21b4e0, 4568cc8cd71f46aeb5c27994ff4f1401; no voters:
I20260927 00:45:26.316272 20619 raft_consensus.cc:2781] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 00:45:26.316361 20619 raft_consensus.cc:482] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 00:45:26.316412 20619 raft_consensus.cc:3037] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:26.316725 20609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.311517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42843 (local address 127.0.98.1:44145)
0927 00:45:26.311624 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.311630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.312359 (+ 729us) server_negotiation.cc:407] Connection header received
0927 00:45:26.312511 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.312514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.312557 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.312635 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 00:45:26.312912 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.313419 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.314697 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.315185 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.315246 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.315286 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.315763 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.316600 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:45:26.317211 20286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26dfc37b6a04361bc3c0b6dbe687a13" candidate_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f886d80c20e14ab7a132b22a4e5ae440" is_pre_election: true
I20260927 00:45:26.317459 20286 raft_consensus.cc:2454] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4568cc8cd71f46aeb5c27994ff4f1401 in term 0.
I20260927 00:45:26.317981 20619 raft_consensus.cc:504] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.318353 20619 leader_election.cc:288] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [CANDIDATE]: Term 1 election: Requested vote from peers f886d80c20e14ab7a132b22a4e5ae440 (127.0.98.1:44145), 3e13b7be681b4041abd625360f21b4e0 (127.0.98.3:44333)
I20260927 00:45:26.318722 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26dfc37b6a04361bc3c0b6dbe687a13" candidate_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e13b7be681b4041abd625360f21b4e0"
I20260927 00:45:26.318856 20548 raft_consensus.cc:3037] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:26.319214 20286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26dfc37b6a04361bc3c0b6dbe687a13" candidate_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f886d80c20e14ab7a132b22a4e5ae440"
I20260927 00:45:26.319545 20286 raft_consensus.cc:3037] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:26.320048 20548 raft_consensus.cc:2454] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4568cc8cd71f46aeb5c27994ff4f1401 in term 1.
I20260927 00:45:26.320329 20350 leader_election.cc:302] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e13b7be681b4041abd625360f21b4e0, 4568cc8cd71f46aeb5c27994ff4f1401; no voters:
I20260927 00:45:26.320479 20619 raft_consensus.cc:2781] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:26.320703 20619 raft_consensus.cc:686] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 1 LEADER]: Becoming Leader. State: Replica: 4568cc8cd71f46aeb5c27994ff4f1401, State: Running, Role: LEADER
I20260927 00:45:26.320812 20286 raft_consensus.cc:2454] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4568cc8cd71f46aeb5c27994ff4f1401 in term 1.
I20260927 00:45:26.321041 20619 consensus_queue.cc:237] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } }
I20260927 00:45:26.323917 20144 catalog_manager.cc:5495] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4568cc8cd71f46aeb5c27994ff4f1401 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4568cc8cd71f46aeb5c27994ff4f1401" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 } health_report { overall_health: UNKNOWN } } }
I20260927 00:45:26.345115 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.341565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.2:33637)
0927 00:45:26.341700 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:26.341704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:26.341718 (+ 14us) server_negotiation.cc:407] Connection header received
0927 00:45:26.341884 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:26.341887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:26.341939 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:26.342048 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:26.342278 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.342763 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.343698 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:26.343920 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:26.344031 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:26.344323 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:26.344723 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:26.344863 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:26.344931 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 00:45:26.344980 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:45:26.350503 20286 raft_consensus.cc:1261] T b26dfc37b6a04361bc3c0b6dbe687a13 P f886d80c20e14ab7a132b22a4e5ae440 [term 1 FOLLOWER]: Refusing update from remote peer 4568cc8cd71f46aeb5c27994ff4f1401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:26.350534 20548 raft_consensus.cc:1261] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 1 FOLLOWER]: Refusing update from remote peer 4568cc8cd71f46aeb5c27994ff4f1401: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:26.351257 20621 consensus_queue.cc:1035] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f886d80c20e14ab7a132b22a4e5ae440" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:26.351437 20619 consensus_queue.cc:1035] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e13b7be681b4041abd625360f21b4e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:26.358218 20624 mvcc.cc:205] Tried to move back new op lower bound from 7333764818324459520 to 7333764818213449728. Current Snapshot: MvccSnapshot[applied={T|T < 7333764818324459520}]
I20260927 00:45:26.358433 20625 mvcc.cc:205] Tried to move back new op lower bound from 7333764818324459520 to 7333764818213449728. Current Snapshot: MvccSnapshot[applied={T|T < 7333764818324459520}]
I20260927 00:45:26.366990 20143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:26.367151 20143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 00:45:26.369424 20143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d9dfbd99b4049dc891616bab5d7b495: Sending DeleteTablet for 3 replicas of tablet b26dfc37b6a04361bc3c0b6dbe687a13
I20260927 00:45:26.370225 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:26 UTC) from {username='slave'} at 127.0.0.1:45812
I20260927 00:45:26.370342 20397 tablet_service.cc:1514] Processing DeleteTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:26 UTC) from {username='slave'} at 127.0.0.1:44946
I20260927 00:45:26.370342 20266 tablet_service.cc:1514] Processing DeleteTablet for tablet b26dfc37b6a04361bc3c0b6dbe687a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 00:45:26 UTC) from {username='slave'} at 127.0.0.1:48486
I20260927 00:45:26.370833 20633 tablet_replica.cc:330] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: stopping tablet replica
I20260927 00:45:26.371116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20204
I20260927 00:45:26.371129 20633 raft_consensus.cc:2229] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 00:45:26.371407 20633 raft_consensus.cc:2258] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:26.372925 20633 ts_tablet_manager.cc:1905] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 00:45:26.374095 20634 tablet_replica.cc:330] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: stopping tablet replica
I20260927 00:45:26.374352 20634 raft_consensus.cc:2229] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 1 LEADER]: Raft consensus shutting down.
I20260927 00:45:26.374676 20634 raft_consensus.cc:2258] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 00:45:26.375612 20633 ts_tablet_manager.cc:1918] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:26.375705 20633 log.cc:1196] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-2/wal/wals/b26dfc37b6a04361bc3c0b6dbe687a13
I20260927 00:45:26.376017 20633 ts_tablet_manager.cc:1939] T b26dfc37b6a04361bc3c0b6dbe687a13 P 3e13b7be681b4041abd625360f21b4e0: Deleting consensus metadata
I20260927 00:45:26.377177 20128 catalog_manager.cc:4841] TS 3e13b7be681b4041abd625360f21b4e0 (127.0.98.3:44333): tablet b26dfc37b6a04361bc3c0b6dbe687a13 (table test-table [id=06c50c6fd17644c1b69bb408b5fecbfd]) successfully deleted
I20260927 00:45:26.378149 20634 ts_tablet_manager.cc:1905] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 00:45:26.379187 20131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44145 (error 108)
I20260927 00:45:26.380170 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.379871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44145 (local address 127.0.0.1:48494)
0927 00:45:26.379998 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.380073 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:45:26.380245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20335
W20260927 00:45:26.380434 20131 catalog_manager.cc:4584] TS f886d80c20e14ab7a132b22a4e5ae440 (127.0.98.1:44145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b26dfc37b6a04361bc3c0b6dbe687a13: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44145: connect: Connection refused (error 111)
I20260927 00:45:26.380568 20634 ts_tablet_manager.cc:1918] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 00:45:26.380635 20634 log.cc:1196] T b26dfc37b6a04361bc3c0b6dbe687a13 P 4568cc8cd71f46aeb5c27994ff4f1401: Deleting WAL directory at /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790469678562534-392-0/minicluster-data/ts-1/wal/wals/b26dfc37b6a04361bc3c0b6dbe687a13
I20260927 00:45:26.384860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20466
I20260927 00:45:26.386276 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:26.385042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33637 (local address 127.0.0.1:44960)
0927 00:45:26.386125 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:26.386186 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":985}
I20260927 00:45:26.390036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20113
2026-09-27T00:45:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1007 ms)
[----------] 6 tests from EncryptionPolicyTest (10147 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:26 dist-test-slave-pkws krb5kdc[20639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:26 dist-test-slave-pkws krb5kdc[20639](info): set up 2 sockets
Sep 27 00:45:26 dist-test-slave-pkws krb5kdc[20639](info): commencing operation
krb5kdc: starting...
W20260927 00:45:27.417065 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:27.535336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:27.642372 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:27.642613 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:27.642657 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:27.646615 20652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:27.646664 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:27.646683 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:27.646699 20652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:27.646714 20652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:27.651698 20652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:27.652575 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:27.653488 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:27.661211 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:27.661314 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:27.661226 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:27.661489 20652 server_base.cc:1034] running on GCE node
I20260927 00:45:27.662055 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:27.663084 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:27.664265 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790469927664244 us; error 45 us; skew 500 ppm
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:27.666958 20652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:27.667901 20652 webserver.cc:460] Webserver started at http://127.0.0.1:41125/ using document root <none> and password file <none>
I20260927 00:45:27.668439 20652 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:27.668529 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:27.668752 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:27.670755 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "44394ecc5be44e9eae9075419c5d1bf8"
format_stamp: "Formatted at 2026-09-27 00:45:27 on dist-test-slave-pkws"
server_key: "0d167e465c99207070f15aa9bef6ed49"
server_key_iv: "dcbbea60999602bd33bac5036c40afce"
server_key_version: "encryptionkey@0"
I20260927 00:45:27.671312 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "44394ecc5be44e9eae9075419c5d1bf8"
format_stamp: "Formatted at 2026-09-27 00:45:27 on dist-test-slave-pkws"
server_key: "0d167e465c99207070f15aa9bef6ed49"
server_key_iv: "dcbbea60999602bd33bac5036c40afce"
server_key_version: "encryptionkey@0"
I20260927 00:45:27.674926 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:27.677214 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:27.678378 20652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:45:27.678511 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "44394ecc5be44e9eae9075419c5d1bf8"
format_stamp: "Formatted at 2026-09-27 00:45:27 on dist-test-slave-pkws"
server_key: "0d167e465c99207070f15aa9bef6ed49"
server_key_iv: "dcbbea60999602bd33bac5036c40afce"
server_key_version: "encryptionkey@0"
I20260927 00:45:27.678619 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:27.712000 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:27.714716 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:27.714915 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:27.721716 20652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33975
I20260927 00:45:27.721740 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33975 every 8 connection(s)
I20260927 00:45:27.722604 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:27.725150 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:27.730286 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8: Bootstrap starting.
I20260927 00:45:27.731012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20652
I20260927 00:45:27.731256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:27.731534 392 external_mini_cluster.cc:1428] Setting key 273c546c76b30a5a5adb708394dcc763
I20260927 00:45:27.732936 20720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:27.733670 20720 log.cc:824] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:27.735684 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8: No bootstrap required, opened a new log
I20260927 00:45:27.739935 20720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260927 00:45:27.740159 20720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:27.740237 20720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44394ecc5be44e9eae9075419c5d1bf8, State: Initialized, Role: FOLLOWER
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:45:27.740728 20720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260927 00:45:27.740875 20720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:27.740960 20720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:27.741068 20720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:27.742174 20720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260927 00:45:27.742532 20720 leader_election.cc:302] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44394ecc5be44e9eae9075419c5d1bf8; no voters:
I20260927 00:45:27.742868 20720 leader_election.cc:288] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:27.742972 20725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:27.743137 20725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [term 1 LEADER]: Becoming Leader. State: Replica: 44394ecc5be44e9eae9075419c5d1bf8, State: Running, Role: LEADER
I20260927 00:45:27.743430 20725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260927 00:45:27.744170 20720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:27.744668 20727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44394ecc5be44e9eae9075419c5d1bf8. Latest consensus state: current_term: 1 leader_uuid: "44394ecc5be44e9eae9075419c5d1bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260927 00:45:27.744794 20727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:27.745118 20726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44394ecc5be44e9eae9075419c5d1bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44394ecc5be44e9eae9075419c5d1bf8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260927 00:45:27.745234 20726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:27.746939 20729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:27.747640 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:27.732633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38641 (local address 127.0.0.1:33975)
0927 00:45:27.733066 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:27.733080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:27.733112 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:45:27.733806 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:27.733827 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:27.734153 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:27.734472 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:27.735359 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:27.736364 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:27.737554 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:27.738019 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:27.738132 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:27.741084 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:27.741094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:27.741106 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:27.741133 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:27.743113 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:27.743821 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:27.743826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:27.743834 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:27.743993 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:27.744415 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:27.744418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:27.744419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:27.745402 (+ 983us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:27.746216 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:27.746419 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 00:45:27.746672 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260927 00:45:27.750205 20729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:27.750648 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:27.755820 20729 catalog_manager.cc:1347] Generated new cluster ID: 856433284fd94a17b77cfbaf5a991d43
I20260927 00:45:27.755877 20729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:27.762054 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:27.752707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39385 (local address 127.0.0.1:33975)
0927 00:45:27.752805 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:27.752809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:27.753077 (+ 268us) server_negotiation.cc:407] Connection header received
0927 00:45:27.753178 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:27.753181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:27.753225 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:27.753309 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:27.754144 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:27.755366 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:27.756568 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:27.756926 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:27.757022 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:27.758682 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:27.758685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:27.758688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:27.758711 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:27.760230 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:27.760734 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:27.760737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:27.760739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:27.760811 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:27.761177 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:27.761180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:27.761182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:27.761360 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:27.761457 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:27.761822 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 00:45:27.761911 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 00:45:27.773218 20729 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:27.774075 20729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:27.782580 20729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44394ecc5be44e9eae9075419c5d1bf8: Generated new TSK 0
I20260927 00:45:27.783414 20729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:27.791568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20652
W20260927 00:45:27.797055 20637 connection.cc:535] client connection to 127.0.0.1:33975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33975): Connection reset by peer (error 104)
2026-09-27T00:45:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20751](info): set up 2 sockets
Sep 27 00:45:27 dist-test-slave-pkws krb5kdc[20751](info): commencing operation
krb5kdc: starting...
W20260927 00:45:28.817099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:28 dist-test-slave-pkws krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:28.933359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:29.039361 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:29.039649 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:29.039723 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:29.043670 20764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:29.043758 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:29.043803 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:29.043859 20764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:29.043895 20764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:29.048974 20764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:29.049954 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:29.050930 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:29.059135 20772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:29.059219 20764 server_base.cc:1034] running on GCE node
W20260927 00:45:29.059154 20770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:29.059154 20769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:29.059933 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:29.060930 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:29.062114 20764 hybrid_clock.cc:648] HybridClock initialized: now 1790469929062096 us; error 31 us; skew 500 ppm
Sep 27 00:45:29 dist-test-slave-pkws krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:29.064622 20764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:29.065511 20764 webserver.cc:460] Webserver started at http://127.0.0.1:37003/ using document root <none> and password file <none>
I20260927 00:45:29.066041 20764 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:29.066102 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:29.066329 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:29.068248 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "96a627ad9b4a4ca7a2ac2700926802d7"
format_stamp: "Formatted at 2026-09-27 00:45:29 on dist-test-slave-pkws"
server_key: "8ef71a50c4c1aa684d3fbe9b40b65c1f"
server_key_iv: "39eb750e7a605ce89c4e398805af61d0"
server_key_version: "encryptionkey@0"
I20260927 00:45:29.068800 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "96a627ad9b4a4ca7a2ac2700926802d7"
format_stamp: "Formatted at 2026-09-27 00:45:29 on dist-test-slave-pkws"
server_key: "8ef71a50c4c1aa684d3fbe9b40b65c1f"
server_key_iv: "39eb750e7a605ce89c4e398805af61d0"
server_key_version: "encryptionkey@0"
I20260927 00:45:29.072319 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:45:29.074658 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:29.075834 20764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 00:45:29.075953 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "96a627ad9b4a4ca7a2ac2700926802d7"
format_stamp: "Formatted at 2026-09-27 00:45:29 on dist-test-slave-pkws"
server_key: "8ef71a50c4c1aa684d3fbe9b40b65c1f"
server_key_iv: "39eb750e7a605ce89c4e398805af61d0"
server_key_version: "encryptionkey@0"
I20260927 00:45:29.076047 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:29.111299 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:29.114061 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:29.114207 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:29.121099 20764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40305
I20260927 00:45:29.121220 20831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40305 every 8 connection(s)
I20260927 00:45:29.122074 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:29.124676 20832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:29.129371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20764
I20260927 00:45:29.129516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:29.129825 392 external_mini_cluster.cc:1428] Setting key a4dd307aeeeb8042671594b16a9c7635
I20260927 00:45:29.130935 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: Bootstrap starting.
I20260927 00:45:29.133219 20832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:29.133935 20832 log.cc:824] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:29.135720 20832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: No bootstrap required, opened a new log
I20260927 00:45:29.138190 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:29.131188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33365 (local address 127.0.0.1:40305)
0927 00:45:29.131581 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:29.131606 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:29.131640 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:45:29.132237 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:29.132253 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:29.132562 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:29.132831 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:29.133196 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:29.134225 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:29.135087 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:29.135561 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:29.135729 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:29.136573 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:29.136579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:29.136590 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:29.137028 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:29.137079 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:29.137101 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:29.137233 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:29.137414 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 00:45:29.137553 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":150,"threads_started":1}
I20260927 00:45:29.139465 20832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260927 00:45:29.139694 20832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:29.139773 20832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a627ad9b4a4ca7a2ac2700926802d7, State: Initialized, Role: FOLLOWER
I20260927 00:45:29.140241 20832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260927 00:45:29.140416 20832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:29.140514 20832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:29.140645 20832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:29.141188 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:29.141664 20832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260927 00:45:29.142014 20832 leader_election.cc:302] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a627ad9b4a4ca7a2ac2700926802d7; no voters:
I20260927 00:45:29.142328 20832 leader_election.cc:288] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:29.142441 20838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:29.142648 20838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [term 1 LEADER]: Becoming Leader. State: Replica: 96a627ad9b4a4ca7a2ac2700926802d7, State: Running, Role: LEADER
I20260927 00:45:29.142930 20838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } }
I20260927 00:45:29.143528 20832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:29.144057 20842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a627ad9b4a4ca7a2ac2700926802d7. Latest consensus state: current_term: 1 leader_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } } }
I20260927 00:45:29.144193 20842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:29.144446 20843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a627ad9b4a4ca7a2ac2700926802d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40305 } } }
I20260927 00:45:29.144536 20843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:29.148386 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:29.143208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35537 (local address 127.0.0.1:40305)
0927 00:45:29.143315 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:29.143319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:29.143872 (+ 553us) server_negotiation.cc:407] Connection header received
0927 00:45:29.144491 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:29.144494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:29.144541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:29.144596 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:29.144994 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:29.145782 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:29.146711 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:29.146995 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:29.147110 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:29.147942 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:29.147946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:29.147949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:29.147968 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:29.147991 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:29.148001 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:29.148098 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:29.148220 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:45:29.148252 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:45:29.148463 20857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:29.148533 20857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:29.148603 20848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:29.149426 20848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:29.154412 20848 catalog_manager.cc:1347] Generated new cluster ID: 2e915d7af9074107ba8f5c5a4dfbfd1e
I20260927 00:45:29.154489 20848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:29.165182 20848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:29.166051 20848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:29.182559 20848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a627ad9b4a4ca7a2ac2700926802d7: Generated new TSK 0
I20260927 00:45:29.183224 20848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:29.189877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20764
W20260927 00:45:29.195313 20749 connection.cc:535] client connection to 127.0.0.1:40305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40305): Connection reset by peer (error 104)
2026-09-27T00:45:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:29 dist-test-slave-pkws krb5kdc[20863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:29 dist-test-slave-pkws krb5kdc[20863](info): set up 2 sockets
Sep 27 00:45:29 dist-test-slave-pkws krb5kdc[20863](info): commencing operation
krb5kdc: starting...
W20260927 00:45:30.265038 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:30 dist-test-slave-pkws krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:30.385782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:30.490765 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:30.490988 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:30.491027 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:30.494889 20876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:30.494937 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:30.494956 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:30.494972 20876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:30.494987 20876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:30.500195 20876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:30.501097 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:30.501965 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:30.509685 20881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:30.509686 20882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:30.509836 20876 server_base.cc:1034] running on GCE node
W20260927 00:45:30.509699 20884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:30.510489 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:30.511540 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:30.512769 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790469930512744 us; error 43 us; skew 500 ppm
Sep 27 00:45:30 dist-test-slave-pkws krb5kdc[20863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:30.515470 20876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:30.516309 20876 webserver.cc:460] Webserver started at http://127.0.0.1:36489/ using document root <none> and password file <none>
I20260927 00:45:30.516815 20876 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:30.516872 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:30.517097 20876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:30.518956 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1"
format_stamp: "Formatted at 2026-09-27 00:45:30 on dist-test-slave-pkws"
server_key: "ab65843ffb07f3f902560fe4dac1b454"
server_key_iv: "534c42886361dcf6bcd1e96a103edb65"
server_key_version: "encryptionkey@0"
I20260927 00:45:30.519447 20876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1"
format_stamp: "Formatted at 2026-09-27 00:45:30 on dist-test-slave-pkws"
server_key: "ab65843ffb07f3f902560fe4dac1b454"
server_key_iv: "534c42886361dcf6bcd1e96a103edb65"
server_key_version: "encryptionkey@0"
I20260927 00:45:30.522938 20876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:30.525141 20891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:30.526309 20876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:45:30.526409 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1"
format_stamp: "Formatted at 2026-09-27 00:45:30 on dist-test-slave-pkws"
server_key: "ab65843ffb07f3f902560fe4dac1b454"
server_key_iv: "534c42886361dcf6bcd1e96a103edb65"
server_key_version: "encryptionkey@0"
I20260927 00:45:30.526489 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:30.530748 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:30.533437 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:30.533568 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:30.539990 20876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35473
I20260927 00:45:30.540015 20943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35473 every 8 connection(s)
I20260927 00:45:30.540830 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:30.543403 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:30.548540 20944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1: Bootstrap starting.
I20260927 00:45:30.550788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20876
I20260927 00:45:30.550942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:30.551014 20944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:30.551294 392 external_mini_cluster.cc:1428] Setting key 814fae15d12dd9d3287c25cef0eb9e7e
I20260927 00:45:30.551848 20944 log.cc:824] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:30.553855 20944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1: No bootstrap required, opened a new log
I20260927 00:45:30.555996 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:30.552377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46905 (local address 127.0.0.1:35473)
0927 00:45:30.552754 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:30.552769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:30.552802 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:30.553479 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:30.553495 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:30.553746 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:30.554081 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:30.554862 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:30.554867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:30.554877 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:30.554887 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:30.554944 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:30.554964 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:30.555078 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:30.555184 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 00:45:30.555340 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":114,"threads_started":1}
I20260927 00:45:30.557585 20944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 00:45:30.557852 20944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:30.557928 20944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8fa8844dd94aa7b8f82cfd3e141cc1, State: Initialized, Role: FOLLOWER
I20260927 00:45:30.558422 20944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 00:45:30.558574 20944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:30.558646 20944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:30.558744 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:30.558748 20944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:30.559592 20944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 00:45:30.559938 20944 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d8fa8844dd94aa7b8f82cfd3e141cc1; no voters:
I20260927 00:45:30.560236 20944 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:30.560330 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:30.560544 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 6d8fa8844dd94aa7b8f82cfd3e141cc1, State: Running, Role: LEADER
I20260927 00:45:30.560869 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } }
I20260927 00:45:30.561451 20944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:30.562252 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8fa8844dd94aa7b8f82cfd3e141cc1. Latest consensus state: current_term: 1 leader_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260927 00:45:30.562393 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:30.563241 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8fa8844dd94aa7b8f82cfd3e141cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35473 } } }
I20260927 00:45:30.563356 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:30.563565 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:30.561121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56057 (local address 127.0.0.1:35473)
0927 00:45:30.561211 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:30.561215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:30.561334 (+ 119us) server_negotiation.cc:407] Connection header received
0927 00:45:30.561430 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:30.561433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:30.561468 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:30.561608 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:30.562827 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:30.562830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:30.562833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:30.562844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:30.562870 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:30.562882 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:30.562966 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:30.563277 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 00:45:30.563312 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 00:45:30.563990 20963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:30.566632 20963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:30.571272 20963 catalog_manager.cc:1347] Generated new cluster ID: be2f3f04b1fd4782af6f344db0a29b3f
I20260927 00:45:30.571331 20963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:30.602775 20963 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:30.603128 20963 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:30.616603 20963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8fa8844dd94aa7b8f82cfd3e141cc1: Generated new TSK 0
I20260927 00:45:30.617424 20963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:30.629161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20876
2026-09-27T00:45:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:30 dist-test-slave-pkws krb5kdc[20975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:30 dist-test-slave-pkws krb5kdc[20975](info): set up 2 sockets
Sep 27 00:45:30 dist-test-slave-pkws krb5kdc[20975](info): commencing operation
krb5kdc: starting...
W20260927 00:45:31.661127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:31 dist-test-slave-pkws krb5kdc[20975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:31.777365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:31.886394 20988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:31.886684 20988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:31.886759 20988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:31.890722 20988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:31.890812 20988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:31.890847 20988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:31.890885 20988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:31.890916 20988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:31.896108 20988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:31.897151 20988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:31.898141 20988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:31.906442 20993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:31.906461 20994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:31.906446 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:31.906790 20988 server_base.cc:1034] running on GCE node
I20260927 00:45:31.907310 20988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:31.908351 20988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:31.909535 20988 hybrid_clock.cc:648] HybridClock initialized: now 1790469931909525 us; error 28 us; skew 500 ppm
Sep 27 00:45:31 dist-test-slave-pkws krb5kdc[20975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:31.917582 20988 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:31.918458 20988 webserver.cc:460] Webserver started at http://127.0.0.1:36559/ using document root <none> and password file <none>
I20260927 00:45:31.919015 20988 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:31.919103 20988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:31.919338 20988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:31.921409 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "b362cf1a585d417c8d15c1237e0f3844"
format_stamp: "Formatted at 2026-09-27 00:45:31 on dist-test-slave-pkws"
server_key: "11825791efaa6fe6388176fca1cf2c4b"
server_key_iv: "058d8c1b9de0800deaf7f687dde72db5"
server_key_version: "encryptionkey@0"
I20260927 00:45:31.921952 20988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "b362cf1a585d417c8d15c1237e0f3844"
format_stamp: "Formatted at 2026-09-27 00:45:31 on dist-test-slave-pkws"
server_key: "11825791efaa6fe6388176fca1cf2c4b"
server_key_iv: "058d8c1b9de0800deaf7f687dde72db5"
server_key_version: "encryptionkey@0"
I20260927 00:45:31.925653 20988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:31.927985 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:31.929329 20988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 00:45:31.929481 20988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "b362cf1a585d417c8d15c1237e0f3844"
format_stamp: "Formatted at 2026-09-27 00:45:31 on dist-test-slave-pkws"
server_key: "11825791efaa6fe6388176fca1cf2c4b"
server_key_iv: "058d8c1b9de0800deaf7f687dde72db5"
server_key_version: "encryptionkey@0"
I20260927 00:45:31.929590 20988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:31.942305 20988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:31.945076 20988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:31.945336 20988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:31.952325 20988 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42783
I20260927 00:45:31.952344 21055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42783 every 8 connection(s)
I20260927 00:45:31.953217 20988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:31.955793 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:31.961019 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844: Bootstrap starting.
I20260927 00:45:31.962872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 20988
I20260927 00:45:31.963001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:31.963274 392 external_mini_cluster.cc:1428] Setting key 3ba87dbbc58045cc12ab5cd68be50661
I20260927 00:45:31.963546 21056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:31.964445 21056 log.cc:824] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:31.966580 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844: No bootstrap required, opened a new log
I20260927 00:45:31.970775 21056 raft_consensus.cc:348] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260927 00:45:31.971014 21056 raft_consensus.cc:374] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:31.971096 21056 raft_consensus.cc:729] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b362cf1a585d417c8d15c1237e0f3844, State: Initialized, Role: FOLLOWER
I20260927 00:45:31.971551 21056 consensus_queue.cc:260] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260927 00:45:31.971693 21056 raft_consensus.cc:388] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:31.971762 21056 raft_consensus.cc:482] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:31.971864 21056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 00:45:31 dist-test-slave-pkws krb5kdc[20975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:45:31.972977 21056 raft_consensus.cc:504] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260927 00:45:31.973310 21056 leader_election.cc:302] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b362cf1a585d417c8d15c1237e0f3844; no voters:
I20260927 00:45:31.973650 21056 leader_election.cc:288] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:31.973752 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:31.974267 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [term 1 LEADER]: Becoming Leader. State: Replica: b362cf1a585d417c8d15c1237e0f3844, State: Running, Role: LEADER
I20260927 00:45:31.974591 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } }
I20260927 00:45:31.974972 21056 sys_catalog.cc:564] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:31.976356 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b362cf1a585d417c8d15c1237e0f3844. Latest consensus state: current_term: 1 leader_uuid: "b362cf1a585d417c8d15c1237e0f3844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } } }
I20260927 00:45:31.976503 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:31.976835 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b362cf1a585d417c8d15c1237e0f3844" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b362cf1a585d417c8d15c1237e0f3844" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42783 } } }
I20260927 00:45:31.976974 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:31.977285 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:31.979142 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:31.964359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40913 (local address 127.0.0.1:42783)
0927 00:45:31.964770 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:31.964794 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:31.964836 (+ 42us) server_negotiation.cc:407] Connection header received
0927 00:45:31.965630 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:31.965653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:31.966071 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:31.966433 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:31.967212 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:31.968713 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:31.969661 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:31.970364 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:31.970519 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:31.973448 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:31.973454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:31.973469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:31.973496 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:31.975752 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:31.976243 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:31.976247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:31.976251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:31.976428 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:31.976796 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:31.976800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:31.976804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:31.977379 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:31.977571 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:31.977969 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 00:45:31.978269 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260927 00:45:31.980679 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:31.981889 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:31.989455 21070 catalog_manager.cc:1347] Generated new cluster ID: f522f7d0403145d7b6761862a1ef9dc6
I20260927 00:45:31.989523 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:31.997264 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:31.984011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39185 (local address 127.0.0.1:42783)
0927 00:45:31.984151 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:31.984155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:31.985459 (+ 1304us) server_negotiation.cc:407] Connection header received
0927 00:45:31.985642 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:31.985645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:31.985694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:31.985781 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:31.986573 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:31.987375 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:31.988410 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:31.988764 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:31.988871 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:31.993925 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:31.993929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:31.993932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:31.993957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:31.995612 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:31.996064 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:31.996067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:31.996069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:31.996150 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:31.996428 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:31.996431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:31.996433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:31.996633 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:31.996755 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:31.996999 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 00:45:31.997109 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 00:45:32.003679 21070 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:32.004987 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:32.009248 21070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b362cf1a585d417c8d15c1237e0f3844: Generated new TSK 0
I20260927 00:45:32.009997 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:32.021287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 20988
2026-09-27T00:45:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:32 dist-test-slave-pkws krb5kdc[21087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:32 dist-test-slave-pkws krb5kdc[21087](info): set up 2 sockets
Sep 27 00:45:32 dist-test-slave-pkws krb5kdc[21087](info): commencing operation
krb5kdc: starting...
W20260927 00:45:33.069170 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:33 dist-test-slave-pkws krb5kdc[21087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:33.188112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:33.294292 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:33.294610 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:33.294680 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:33.298715 21100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:33.298807 21100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:33.298853 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:33.298938 21100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:33.298978 21100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:33.304204 21100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:33.305323 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:33.306322 21100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:33.314621 21105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:33.314757 21108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:33.314659 21100 server_base.cc:1034] running on GCE node
W20260927 00:45:33.314666 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:33.315332 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:33.316468 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:33.317652 21100 hybrid_clock.cc:648] HybridClock initialized: now 1790469933317642 us; error 29 us; skew 500 ppm
Sep 27 00:45:33 dist-test-slave-pkws krb5kdc[21087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:33.320295 21100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:33.321188 21100 webserver.cc:460] Webserver started at http://127.0.0.1:39503/ using document root <none> and password file <none>
I20260927 00:45:33.321734 21100 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:33.321823 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:33.322050 21100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:33.324043 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "e626fa5d3e8e448bb88a57aeedf16900"
format_stamp: "Formatted at 2026-09-27 00:45:33 on dist-test-slave-pkws"
server_key: "e9472d50e8e6881a4171e13d222d542e"
server_key_iv: "c58e106a8e45f0d943bd2bf89a4f4088"
server_key_version: "encryptionkey@0"
I20260927 00:45:33.324637 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "e626fa5d3e8e448bb88a57aeedf16900"
format_stamp: "Formatted at 2026-09-27 00:45:33 on dist-test-slave-pkws"
server_key: "e9472d50e8e6881a4171e13d222d542e"
server_key_iv: "c58e106a8e45f0d943bd2bf89a4f4088"
server_key_version: "encryptionkey@0"
I20260927 00:45:33.328184 21100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:45:33.330495 21115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:33.331699 21100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:33.331833 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "e626fa5d3e8e448bb88a57aeedf16900"
format_stamp: "Formatted at 2026-09-27 00:45:33 on dist-test-slave-pkws"
server_key: "e9472d50e8e6881a4171e13d222d542e"
server_key_iv: "c58e106a8e45f0d943bd2bf89a4f4088"
server_key_version: "encryptionkey@0"
I20260927 00:45:33.331948 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:33.358610 21100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:33.361608 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:33.361812 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:33.368494 21100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41543
I20260927 00:45:33.368553 21167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41543 every 8 connection(s)
I20260927 00:45:33.369510 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:33.372148 21168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:33.373962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21100
I20260927 00:45:33.374130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:33.374392 392 external_mini_cluster.cc:1428] Setting key c36d077ac2cca2306b5bcb1708077e04
I20260927 00:45:33.378929 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900: Bootstrap starting.
I20260927 00:45:33.381425 21168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:33.382171 21168 log.cc:824] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:33.384110 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:33.375753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45863 (local address 127.0.0.1:41543)
0927 00:45:33.376170 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:33.376201 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:33.376242 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:45:33.377090 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:33.377113 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:33.377540 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:33.377899 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:33.378309 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:33.379701 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:33.380534 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:33.381233 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:33.381420 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:33.382206 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:33.382212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:33.382227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:33.382696 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:33.382746 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:33.382768 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:33.382845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:33.383030 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 00:45:33.383224 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260927 00:45:33.384289 21168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900: No bootstrap required, opened a new log
I20260927 00:45:33.386826 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:33.388146 21168 raft_consensus.cc:348] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } }
I20260927 00:45:33.388382 21168 raft_consensus.cc:374] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:33.388435 21168 raft_consensus.cc:729] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e626fa5d3e8e448bb88a57aeedf16900, State: Initialized, Role: FOLLOWER
I20260927 00:45:33.388947 21168 consensus_queue.cc:260] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } }
I20260927 00:45:33.389102 21168 raft_consensus.cc:388] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:33.389173 21168 raft_consensus.cc:482] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:33.389281 21168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:33.390254 21168 raft_consensus.cc:504] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } }
I20260927 00:45:33.390606 21168 leader_election.cc:302] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e626fa5d3e8e448bb88a57aeedf16900; no voters:
I20260927 00:45:33.390910 21168 leader_election.cc:288] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:33.391029 21178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:33.391252 21178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [term 1 LEADER]: Becoming Leader. State: Replica: e626fa5d3e8e448bb88a57aeedf16900, State: Running, Role: LEADER
I20260927 00:45:33.391611 21178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } }
I20260927 00:45:33.392289 21168 sys_catalog.cc:564] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:33.392828 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:33.388652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55773 (local address 127.0.0.1:41543)
0927 00:45:33.388764 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:33.388768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:33.388942 (+ 174us) server_negotiation.cc:407] Connection header received
0927 00:45:33.389093 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:33.389096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:33.389135 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:33.389233 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:33.389548 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:33.390031 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:33.391111 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:33.391474 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:33.391575 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:33.392387 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:33.392391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:33.392394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:33.392407 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:33.392442 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:33.392455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:33.392529 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:33.392636 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 00:45:33.392701 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:33.393520 21180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e626fa5d3e8e448bb88a57aeedf16900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } } }
I20260927 00:45:33.392866 21178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e626fa5d3e8e448bb88a57aeedf16900. Latest consensus state: current_term: 1 leader_uuid: "e626fa5d3e8e448bb88a57aeedf16900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e626fa5d3e8e448bb88a57aeedf16900" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41543 } } }
I20260927 00:45:33.393755 21178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:33.394014 21180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:33.394094 21187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:33.397207 21187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:33.402802 21187 catalog_manager.cc:1347] Generated new cluster ID: eafab69e80674797b34c70363c7c031e
I20260927 00:45:33.402884 21187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:33.417151 21187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:33.418084 21187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:33.423504 21187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e626fa5d3e8e448bb88a57aeedf16900: Generated new TSK 0
I20260927 00:45:33.424162 21187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:33.432351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21100
2026-09-27T00:45:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:33 dist-test-slave-pkws krb5kdc[21199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:33 dist-test-slave-pkws krb5kdc[21199](info): set up 2 sockets
Sep 27 00:45:33 dist-test-slave-pkws krb5kdc[21199](info): commencing operation
krb5kdc: starting...
W20260927 00:45:34.465085 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:34 dist-test-slave-pkws krb5kdc[21199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:34.578212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:34.683805 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:34.684079 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:34.684151 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:34.688063 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:34.688146 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:34.688184 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:34.688241 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:34.688285 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:34.693385 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:34.694339 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:34.695290 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:34.703505 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:34.703644 21212 server_base.cc:1034] running on GCE node
W20260927 00:45:34.703490 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:34.703490 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:34.704344 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:34.705391 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:34.706578 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790469934706554 us; error 37 us; skew 500 ppm
Sep 27 00:45:34 dist-test-slave-pkws krb5kdc[21199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:34.709172 21212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:34.710022 21212 webserver.cc:460] Webserver started at http://127.0.0.1:44315/ using document root <none> and password file <none>
I20260927 00:45:34.710547 21212 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:34.710638 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:34.710860 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:34.712751 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "5c458a142ccb490c926aa394f4d81841"
format_stamp: "Formatted at 2026-09-27 00:45:34 on dist-test-slave-pkws"
server_key: "14d019ad35bf8192dad4a86eeef008cb"
server_key_iv: "440ef23ed8d1a1ffc1c0ae49a61f67d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:34.713366 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c458a142ccb490c926aa394f4d81841"
format_stamp: "Formatted at 2026-09-27 00:45:34 on dist-test-slave-pkws"
server_key: "14d019ad35bf8192dad4a86eeef008cb"
server_key_iv: "440ef23ed8d1a1ffc1c0ae49a61f67d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:34.716744 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:45:34.718971 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:34.720098 21212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:34.720224 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "5c458a142ccb490c926aa394f4d81841"
format_stamp: "Formatted at 2026-09-27 00:45:34 on dist-test-slave-pkws"
server_key: "14d019ad35bf8192dad4a86eeef008cb"
server_key_iv: "440ef23ed8d1a1ffc1c0ae49a61f67d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:34.720345 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:34.724880 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:34.727556 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:34.727742 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:34.734556 21212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44873
I20260927 00:45:34.734581 21279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44873 every 8 connection(s)
I20260927 00:45:34.735417 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:34.738081 21280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:34.743319 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841: Bootstrap starting.
I20260927 00:45:34.743479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21212
I20260927 00:45:34.743637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:34.743942 392 external_mini_cluster.cc:1428] Setting key 3efa33871f95abb8f0fe8244c4da22e1
I20260927 00:45:34.745956 21280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:34.746657 21280 log.cc:824] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:34.748723 21280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841: No bootstrap required, opened a new log
I20260927 00:45:34.749168 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:34.745321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33079 (local address 127.0.0.1:44873)
0927 00:45:34.745726 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:34.745740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:34.745774 (+ 34us) server_negotiation.cc:407] Connection header received
0927 00:45:34.746411 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:34.746428 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:34.746691 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:34.747054 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:34.747844 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:34.747850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:34.747864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:34.747875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:34.747932 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:34.747956 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:34.748064 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:34.748153 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:45:34.748299 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20260927 00:45:34.751681 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:34.752410 21280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260927 00:45:34.752676 21280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:34.752740 21280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c458a142ccb490c926aa394f4d81841, State: Initialized, Role: FOLLOWER
I20260927 00:45:34.753224 21280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260927 00:45:34.753379 21280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:34.753449 21280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:34.753545 21280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:34.754681 21280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260927 00:45:34.755038 21280 leader_election.cc:302] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c458a142ccb490c926aa394f4d81841; no voters:
I20260927 00:45:34.755373 21280 leader_election.cc:288] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:34.755468 21290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:34.755673 21290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [term 1 LEADER]: Becoming Leader. State: Replica: 5c458a142ccb490c926aa394f4d81841, State: Running, Role: LEADER
I20260927 00:45:34.755705 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:34.753945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46397 (local address 127.0.0.1:44873)
0927 00:45:34.754042 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:34.754046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:34.754273 (+ 227us) server_negotiation.cc:407] Connection header received
0927 00:45:34.754394 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:34.754396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:34.754423 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:34.754487 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:34.755216 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:34.755219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:34.755222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:34.755234 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:34.755265 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:34.755276 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:34.755364 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:34.755529 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 00:45:34.755574 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 00:45:34.756230 21290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } }
I20260927 00:45:34.756618 21280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:34.757500 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c458a142ccb490c926aa394f4d81841. Latest consensus state: current_term: 1 leader_uuid: "5c458a142ccb490c926aa394f4d81841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } } }
I20260927 00:45:34.757628 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:34.757884 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c458a142ccb490c926aa394f4d81841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c458a142ccb490c926aa394f4d81841" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44873 } } }
I20260927 00:45:34.757972 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:34.758106 21299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:34.761139 21299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:34.766860 21299 catalog_manager.cc:1347] Generated new cluster ID: 952becd274444874be408804d028f154
I20260927 00:45:34.766932 21299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:34.784238 21299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:34.784569 21299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:34.789106 21299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c458a142ccb490c926aa394f4d81841: Generated new TSK 0
I20260927 00:45:34.789681 21299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:34.795796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21212
2026-09-27T00:45:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:34 dist-test-slave-pkws krb5kdc[21311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:34 dist-test-slave-pkws krb5kdc[21311](info): set up 2 sockets
Sep 27 00:45:34 dist-test-slave-pkws krb5kdc[21311](info): commencing operation
krb5kdc: starting...
W20260927 00:45:35.825378 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:35 dist-test-slave-pkws krb5kdc[21311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:35.947973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:36.062052 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:36.062497 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:36.062603 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:36.067677 21324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:36.067764 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:36.067811 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:36.067847 21324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:36.067883 21324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:36.073954 21324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:36.075024 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:36.076020 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:36.084352 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:36.084368 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:36.084368 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:36.084647 21324 server_base.cc:1034] running on GCE node
I20260927 00:45:36.085256 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:36.086315 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:36.087502 21324 hybrid_clock.cc:648] HybridClock initialized: now 1790469936087476 us; error 44 us; skew 500 ppm
Sep 27 00:45:36 dist-test-slave-pkws krb5kdc[21311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:36.090207 21324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:36.091006 21324 webserver.cc:460] Webserver started at http://127.0.0.1:34369/ using document root <none> and password file <none>
I20260927 00:45:36.091535 21324 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:36.091594 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:36.091833 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:36.093860 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "2b3b46ad08b74bfc9081f5e50804a414"
format_stamp: "Formatted at 2026-09-27 00:45:36 on dist-test-slave-pkws"
server_key: "243f16bbe466ecb521307015ac97c1cb"
server_key_iv: "2be55efd8e232b4c4fa6a0bbe56bd9c6"
server_key_version: "encryptionkey@0"
I20260927 00:45:36.094395 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b3b46ad08b74bfc9081f5e50804a414"
format_stamp: "Formatted at 2026-09-27 00:45:36 on dist-test-slave-pkws"
server_key: "243f16bbe466ecb521307015ac97c1cb"
server_key_iv: "2be55efd8e232b4c4fa6a0bbe56bd9c6"
server_key_version: "encryptionkey@0"
I20260927 00:45:36.097872 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:36.100065 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:36.101244 21324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:36.101369 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "2b3b46ad08b74bfc9081f5e50804a414"
format_stamp: "Formatted at 2026-09-27 00:45:36 on dist-test-slave-pkws"
server_key: "243f16bbe466ecb521307015ac97c1cb"
server_key_iv: "2be55efd8e232b4c4fa6a0bbe56bd9c6"
server_key_version: "encryptionkey@0"
I20260927 00:45:36.101466 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:36.115731 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:36.118842 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:36.119035 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:36.125690 21324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36313
I20260927 00:45:36.125712 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36313 every 8 connection(s)
I20260927 00:45:36.126588 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:36.129320 21392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:36.133505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21324
I20260927 00:45:36.133651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:36.133947 392 external_mini_cluster.cc:1428] Setting key 0e153c91ce4cc69f0b1a5a3f86bdebe1
I20260927 00:45:36.134835 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: Bootstrap starting.
I20260927 00:45:36.137511 21392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:36.138327 21392 log.cc:824] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:36.140209 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: No bootstrap required, opened a new log
Sep 27 00:45:36 dist-test-slave-pkws krb5kdc[21311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:45:36.143559 21392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:45:36.143836 21392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:36.143949 21392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3b46ad08b74bfc9081f5e50804a414, State: Initialized, Role: FOLLOWER
I20260927 00:45:36.144433 21392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:45:36.144622 21392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:36.144712 21392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:36.144826 21392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:36.145654 21392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:45:36.146037 21392 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3b46ad08b74bfc9081f5e50804a414; no voters:
I20260927 00:45:36.146365 21392 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:36.146476 21397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:36.146721 21397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3b46ad08b74bfc9081f5e50804a414, State: Running, Role: LEADER
I20260927 00:45:36.147075 21397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } }
I20260927 00:45:36.147626 21392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:36.148226 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3b46ad08b74bfc9081f5e50804a414. Latest consensus state: current_term: 1 leader_uuid: "2b3b46ad08b74bfc9081f5e50804a414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260927 00:45:36.148646 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3b46ad08b74bfc9081f5e50804a414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3b46ad08b74bfc9081f5e50804a414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36313 } } }
I20260927 00:45:36.148784 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:36.149125 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:36.135301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:49935 (local address 127.0.0.1:36313)
0927 00:45:36.135685 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:36.135703 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:36.135743 (+ 40us) server_negotiation.cc:407] Connection header received
0927 00:45:36.136588 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:36.136610 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:36.137053 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:36.137392 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:36.138133 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:36.139520 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:36.140379 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:36.140831 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:36.141052 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:36.143434 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:36.143439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:36.143452 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:36.143476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:36.145377 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:36.145934 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:36.145939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:36.145945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:36.146113 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:36.146614 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:36.146617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:36.146619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:36.147075 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:36.147330 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:36.147647 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 00:45:36.148050 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260927 00:45:36.148401 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:36.152007 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260927 00:45:36.152384 21412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:36.152472 21412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:36.152539 21406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:36.153363 21406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:36.158414 21406 catalog_manager.cc:1347] Generated new cluster ID: 8c920732665d407f958956cfc4303734
I20260927 00:45:36.158468 21406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:36.163091 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:36.153986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:54521 (local address 127.0.0.1:36313)
0927 00:45:36.154089 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:36.154093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:36.154318 (+ 225us) server_negotiation.cc:407] Connection header received
0927 00:45:36.154420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:36.154422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:36.154456 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:36.154534 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:36.155320 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:36.156038 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:36.157229 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:36.157562 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:36.157663 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:36.159291 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:36.159295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:36.159297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:36.159322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:36.161019 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:36.161630 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:36.161634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:36.161635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:36.161712 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:36.162089 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:36.162092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:36.162094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:36.162289 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:36.162401 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:36.162855 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 00:45:36.162947 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 00:45:36.170909 21406 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:36.171679 21406 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:36.179452 21406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3b46ad08b74bfc9081f5e50804a414: Generated new TSK 0
I20260927 00:45:36.180068 21406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:36.186723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21324
W20260927 00:45:36.191917 21309 connection.cc:535] client connection to 127.0.0.1:36313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36313): Connection reset by peer (error 104)
2026-09-27T00:45:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:36 dist-test-slave-pkws krb5kdc[21423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:36 dist-test-slave-pkws krb5kdc[21423](info): set up 2 sockets
Sep 27 00:45:36 dist-test-slave-pkws krb5kdc[21423](info): commencing operation
krb5kdc: starting...
W20260927 00:45:37.225068 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:37 dist-test-slave-pkws krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:37.339340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:37.448130 21436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:37.448375 21436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:37.448415 21436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:37.452425 21436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:37.452493 21436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:37.452512 21436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:37.452529 21436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:37.452582 21436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:37.457810 21436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:37.458799 21436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:37.459770 21436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:37.468025 21442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:37.468070 21444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:37.468025 21436 server_base.cc:1034] running on GCE node
W20260927 00:45:37.468070 21441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:37.468833 21436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:37.469864 21436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:37.471020 21436 hybrid_clock.cc:648] HybridClock initialized: now 1790469937470988 us; error 44 us; skew 500 ppm
Sep 27 00:45:37 dist-test-slave-pkws krb5kdc[21423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:37.479044 21436 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:37.479889 21436 webserver.cc:460] Webserver started at http://127.0.0.1:40185/ using document root <none> and password file <none>
I20260927 00:45:37.480424 21436 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:37.480522 21436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:37.480746 21436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:37.482717 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "7086d302d2eb4c7f9ab40edd99d59570"
format_stamp: "Formatted at 2026-09-27 00:45:37 on dist-test-slave-pkws"
server_key: "4b39f96a276acba67a83b1e2f5bda912"
server_key_iv: "7c691218e0932a09e35bd71581ecaed6"
server_key_version: "encryptionkey@0"
I20260927 00:45:37.483264 21436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "7086d302d2eb4c7f9ab40edd99d59570"
format_stamp: "Formatted at 2026-09-27 00:45:37 on dist-test-slave-pkws"
server_key: "4b39f96a276acba67a83b1e2f5bda912"
server_key_iv: "7c691218e0932a09e35bd71581ecaed6"
server_key_version: "encryptionkey@0"
I20260927 00:45:37.486759 21436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:37.489035 21451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:37.490128 21436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:37.490272 21436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "7086d302d2eb4c7f9ab40edd99d59570"
format_stamp: "Formatted at 2026-09-27 00:45:37 on dist-test-slave-pkws"
server_key: "4b39f96a276acba67a83b1e2f5bda912"
server_key_iv: "7c691218e0932a09e35bd71581ecaed6"
server_key_version: "encryptionkey@0"
I20260927 00:45:37.490382 21436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:37.507391 21436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:37.510463 21436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:37.510671 21436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:37.517503 21436 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40159
I20260927 00:45:37.517519 21503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40159 every 8 connection(s)
I20260927 00:45:37.518337 21436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:37.521013 21504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:37.525027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21436
I20260927 00:45:37.525164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:37.525481 392 external_mini_cluster.cc:1428] Setting key 6113d3400d40e18c50a99bc8df978338
I20260927 00:45:37.526393 21504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570: Bootstrap starting.
I20260927 00:45:37.528551 21504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:37.529337 21504 log.cc:824] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:37.531095 21504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570: No bootstrap required, opened a new log
I20260927 00:45:37.534060 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:37.526776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:41875 (local address 127.0.0.1:40159)
0927 00:45:37.527188 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:37.527202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:37.527234 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:45:37.527832 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:37.527847 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:37.528174 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:37.528491 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:37.528880 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:37.530026 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:37.530907 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:37.531422 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:37.531610 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:37.532462 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:37.532467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:37.532478 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:37.532837 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:37.532930 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:37.532977 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:37.533111 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:37.533286 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 00:45:37.533413 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260927 00:45:37.534652 21504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260927 00:45:37.534878 21504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:37.535583 21504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7086d302d2eb4c7f9ab40edd99d59570, State: Initialized, Role: FOLLOWER
I20260927 00:45:37.536075 21504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260927 00:45:37.536252 21504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:37.536379 21504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:37.536499 21504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:37.536834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:37.537426 21504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260927 00:45:37.537742 21504 leader_election.cc:302] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7086d302d2eb4c7f9ab40edd99d59570; no voters:
I20260927 00:45:37.538067 21504 leader_election.cc:288] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:37.538160 21511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:37.538401 21511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [term 1 LEADER]: Becoming Leader. State: Replica: 7086d302d2eb4c7f9ab40edd99d59570, State: Running, Role: LEADER
I20260927 00:45:37.538772 21511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } }
I20260927 00:45:37.539338 21504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:37.539979 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7086d302d2eb4c7f9ab40edd99d59570. Latest consensus state: current_term: 1 leader_uuid: "7086d302d2eb4c7f9ab40edd99d59570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } } }
I20260927 00:45:37.540107 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:37.540764 21515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7086d302d2eb4c7f9ab40edd99d59570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7086d302d2eb4c7f9ab40edd99d59570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40159 } } }
I20260927 00:45:37.540943 21515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:37.541848 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:37.543697 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:37.538835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:57033 (local address 127.0.0.1:40159)
0927 00:45:37.538928 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:37.538931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:37.539242 (+ 311us) server_negotiation.cc:407] Connection header received
0927 00:45:37.539408 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:37.539411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:37.539464 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:37.539545 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:37.539958 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:37.540776 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:37.542049 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:37.542337 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:37.542452 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:37.543211 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:37.543215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:37.543217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:37.543229 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:37.543260 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:37.543270 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:37.543371 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:37.543518 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 00:45:37.543550 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 00:45:37.545050 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:37.549926 21521 catalog_manager.cc:1347] Generated new cluster ID: 901d42857c8e414a944e3d042dae6806
I20260927 00:45:37.549988 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:37.563483 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:37.564356 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:37.573369 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7086d302d2eb4c7f9ab40edd99d59570: Generated new TSK 0
I20260927 00:45:37.573978 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:37.580375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21436
W20260927 00:45:37.585534 21421 connection.cc:535] client connection to 127.0.0.1:40159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40159): Connection reset by peer (error 104)
2026-09-27T00:45:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:37 dist-test-slave-pkws krb5kdc[21535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:37 dist-test-slave-pkws krb5kdc[21535](info): set up 2 sockets
Sep 27 00:45:37 dist-test-slave-pkws krb5kdc[21535](info): commencing operation
krb5kdc: starting...
W20260927 00:45:38.617035 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:38 dist-test-slave-pkws krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:38.734190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:38.839820 21548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:38.840044 21548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:38.840082 21548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:38.844053 21548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:38.844106 21548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:38.844125 21548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:38.844141 21548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:38.844172 21548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:38.849434 21548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:38.850363 21548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:38.851307 21548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:38.859341 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:38.859505 21548 server_base.cc:1034] running on GCE node
W20260927 00:45:38.859334 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:38.859355 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:38.860072 21548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:38.861179 21548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:38.862337 21548 hybrid_clock.cc:648] HybridClock initialized: now 1790469938862311 us; error 38 us; skew 500 ppm
Sep 27 00:45:38 dist-test-slave-pkws krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:38.864873 21548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:38.865726 21548 webserver.cc:460] Webserver started at http://127.0.0.1:42657/ using document root <none> and password file <none>
I20260927 00:45:38.866263 21548 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:38.866351 21548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:38.866575 21548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:38.868510 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "96c6fddfc40444008263548bfe71d983"
format_stamp: "Formatted at 2026-09-27 00:45:38 on dist-test-slave-pkws"
server_key: "a44df91b7ceb89232088932beaf0f576"
server_key_iv: "048baf557913453bd18931d8a6734f62"
server_key_version: "encryptionkey@0"
I20260927 00:45:38.869122 21548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "96c6fddfc40444008263548bfe71d983"
format_stamp: "Formatted at 2026-09-27 00:45:38 on dist-test-slave-pkws"
server_key: "a44df91b7ceb89232088932beaf0f576"
server_key_iv: "048baf557913453bd18931d8a6734f62"
server_key_version: "encryptionkey@0"
I20260927 00:45:38.872696 21548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 00:45:38.874987 21563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:38.876148 21548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:38.876302 21548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "96c6fddfc40444008263548bfe71d983"
format_stamp: "Formatted at 2026-09-27 00:45:38 on dist-test-slave-pkws"
server_key: "a44df91b7ceb89232088932beaf0f576"
server_key_iv: "048baf557913453bd18931d8a6734f62"
server_key_version: "encryptionkey@0"
I20260927 00:45:38.876412 21548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:38.880702 21548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:38.883355 21548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:38.883545 21548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:38.890061 21548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44961
I20260927 00:45:38.890074 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44961 every 8 connection(s)
I20260927 00:45:38.890913 21548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:38.893509 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:38.898571 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983: Bootstrap starting.
I20260927 00:45:38.899500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21548
I20260927 00:45:38.899644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:38.899926 392 external_mini_cluster.cc:1428] Setting key 8e67d33156c1a3090aa2b901c0dadf5c
I20260927 00:45:38.901186 21616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:38.901898 21616 log.cc:824] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:38.903927 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983: No bootstrap required, opened a new log
I20260927 00:45:38.904587 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:38.900964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:54097 (local address 127.0.0.1:44961)
0927 00:45:38.901338 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:38.901351 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:38.901383 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:45:38.901998 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:38.902041 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:38.902322 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:38.902628 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:38.903292 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:38.903297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:38.903311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:38.903322 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:38.903380 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:38.903405 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:38.903514 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:38.903602 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 00:45:38.903747 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260927 00:45:38.907107 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:38.908353 21616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260927 00:45:38.908573 21616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:38.908643 21616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c6fddfc40444008263548bfe71d983, State: Initialized, Role: FOLLOWER
I20260927 00:45:38.909150 21616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260927 00:45:38.909309 21616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:38.909373 21616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:38.909473 21616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:38.910558 21616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260927 00:45:38.910775 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:38.909091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:37113 (local address 127.0.0.1:44961)
0927 00:45:38.909223 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:38.909227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:38.909494 (+ 267us) server_negotiation.cc:407] Connection header received
0927 00:45:38.909536 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:38.909538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:38.909590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:38.909660 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:38.910380 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:38.910383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:38.910386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:38.910397 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:38.910423 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:38.910441 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:38.910528 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:38.910599 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:45:38.910639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 00:45:38.910939 21616 leader_election.cc:302] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96c6fddfc40444008263548bfe71d983; no voters:
I20260927 00:45:38.911283 21616 leader_election.cc:288] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:38.911352 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:38.911547 21626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [term 1 LEADER]: Becoming Leader. State: Replica: 96c6fddfc40444008263548bfe71d983, State: Running, Role: LEADER
I20260927 00:45:38.911873 21626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } }
I20260927 00:45:38.912541 21616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:38.913694 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96c6fddfc40444008263548bfe71d983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } } }
I20260927 00:45:38.913691 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c6fddfc40444008263548bfe71d983. Latest consensus state: current_term: 1 leader_uuid: "96c6fddfc40444008263548bfe71d983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c6fddfc40444008263548bfe71d983" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44961 } } }
I20260927 00:45:38.913856 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:38.913859 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:38.914306 21635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:38.917078 21635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:38.921777 21635 catalog_manager.cc:1347] Generated new cluster ID: 7b525158297640729a27ec0b03842e30
I20260927 00:45:38.921835 21635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:38.953270 21635 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:38.953600 21635 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:38.958875 21635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96c6fddfc40444008263548bfe71d983: Generated new TSK 0
I20260927 00:45:38.959486 21635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:38.965531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21548
2026-09-27T00:45:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:39 dist-test-slave-pkws krb5kdc[21647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:39 dist-test-slave-pkws krb5kdc[21647](info): set up 2 sockets
Sep 27 00:45:39 dist-test-slave-pkws krb5kdc[21647](info): commencing operation
krb5kdc: starting...
W20260927 00:45:40.017189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:40.131779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35689
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:40.236351 21660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:40.236616 21660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:40.236685 21660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:40.241237 21660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:40.241308 21660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:40.241353 21660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:40.241395 21660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:40.241451 21660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:40.246645 21660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:40.247565 21660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:40.248531 21660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:40.256527 21666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:40.256546 21665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:40.256769 21660 server_base.cc:1034] running on GCE node
W20260927 00:45:40.256994 21668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:40.257553 21660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:40.258541 21660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:40.259701 21660 hybrid_clock.cc:648] HybridClock initialized: now 1790469940259687 us; error 23 us; skew 500 ppm
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:40.262245 21660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:40.262984 21660 webserver.cc:460] Webserver started at http://127.0.0.1:34657/ using document root <none> and password file <none>
I20260927 00:45:40.263463 21660 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:40.263516 21660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:40.263664 21660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:40.265627 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "0a2c9e71aef240d8826501499729ddd1"
format_stamp: "Formatted at 2026-09-27 00:45:40 on dist-test-slave-pkws"
server_key: "ac88629367ab6ed712c85410c72de30b"
server_key_iv: "08a86176b4b07a33c0387b78c6bb02df"
server_key_version: "encryptionkey@0"
I20260927 00:45:40.266112 21660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a2c9e71aef240d8826501499729ddd1"
format_stamp: "Formatted at 2026-09-27 00:45:40 on dist-test-slave-pkws"
server_key: "ac88629367ab6ed712c85410c72de30b"
server_key_iv: "08a86176b4b07a33c0387b78c6bb02df"
server_key_version: "encryptionkey@0"
I20260927 00:45:40.269650 21660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:40.271816 21675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:40.273041 21660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:40.273155 21660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "0a2c9e71aef240d8826501499729ddd1"
format_stamp: "Formatted at 2026-09-27 00:45:40 on dist-test-slave-pkws"
server_key: "ac88629367ab6ed712c85410c72de30b"
server_key_iv: "08a86176b4b07a33c0387b78c6bb02df"
server_key_version: "encryptionkey@0"
I20260927 00:45:40.273252 21660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:40.289489 21660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:40.291965 21660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:40.292151 21660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:40.298730 21660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35689
I20260927 00:45:40.298815 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35689 every 8 connection(s)
I20260927 00:45:40.299644 21660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:40.302313 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:40.306991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21660
I20260927 00:45:40.307137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:40.307528 392 external_mini_cluster.cc:1428] Setting key 86a248b94d8144fd38e27e3aed07c921
I20260927 00:45:40.307685 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1: Bootstrap starting.
I20260927 00:45:40.310280 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:40.311002 21728 log.cc:824] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:40.312716 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1: No bootstrap required, opened a new log
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 00:45:40.316341 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } }
I20260927 00:45:40.316568 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:40.316633 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2c9e71aef240d8826501499729ddd1, State: Initialized, Role: FOLLOWER
I20260927 00:45:40.317148 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } }
I20260927 00:45:40.317327 21728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:40.317402 21728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:40.317502 21728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:40.318406 21728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } }
I20260927 00:45:40.318742 21728 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2c9e71aef240d8826501499729ddd1; no voters:
I20260927 00:45:40.319059 21728 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:40.319118 21733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:40.319329 21733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2c9e71aef240d8826501499729ddd1, State: Running, Role: LEADER
I20260927 00:45:40.319664 21733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } }
I20260927 00:45:40.320282 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:40.321489 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2c9e71aef240d8826501499729ddd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } } }
I20260927 00:45:40.321501 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2c9e71aef240d8826501499729ddd1. Latest consensus state: current_term: 1 leader_uuid: "0a2c9e71aef240d8826501499729ddd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2c9e71aef240d8826501499729ddd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35689 } } }
I20260927 00:45:40.321691 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:40.321875 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:40.309168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:60607 (local address 127.0.0.1:35689)
0927 00:45:40.309559 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:40.309573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:40.309605 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:45:40.310206 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:40.310221 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:40.310590 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:40.310917 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:40.311707 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:40.312648 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:40.313577 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:40.314030 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:40.314198 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:40.316684 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:40.316690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:40.316706 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:40.316737 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:40.318557 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:40.319136 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:40.319144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:40.319151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:40.319291 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:40.319697 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:40.319700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:40.319702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:40.320098 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:40.320326 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:40.320603 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:45:40.320839 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
I20260927 00:45:40.322067 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:40.322381 21742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:40.324922 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:40.325940 21742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:40.331277 21742 catalog_manager.cc:1347] Generated new cluster ID: 46994b5799a949ffb9e7821d9192324f
I20260927 00:45:40.331332 21742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:40.335814 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:40.327191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:38035 (local address 127.0.0.1:35689)
0927 00:45:40.327291 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:40.327295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:40.327310 (+ 15us) server_negotiation.cc:407] Connection header received
0927 00:45:40.327394 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:40.327400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:40.327440 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:40.327519 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:40.328327 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:40.329063 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:40.330214 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:40.330564 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:40.330666 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:40.332358 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:40.332361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:40.332364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:40.332387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:40.334017 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:40.334550 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:40.334553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:40.334554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:40.334631 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:40.335000 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:40.335003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:40.335004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:40.335182 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:40.335305 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:40.335589 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 00:45:40.335673 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:45:40.343308 21742 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:40.344120 21742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:40.347963 21742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2c9e71aef240d8826501499729ddd1: Generated new TSK 0
I20260927 00:45:40.348537 21742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:40.356338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21660
W20260927 00:45:40.361688 21645 connection.cc:535] client connection to 127.0.0.1:35689 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35689): Connection reset by peer (error 104)
2026-09-27T00:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21759](info): set up 2 sockets
Sep 27 00:45:40 dist-test-slave-pkws krb5kdc[21759](info): commencing operation
krb5kdc: starting...
W20260927 00:45:41.389164 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:41 dist-test-slave-pkws krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:41.507695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:41.620165 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:41.620400 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:41.620440 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:41.624496 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:41.624548 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:41.624568 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:41.624609 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:41.624642 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:41.629639 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:41.630559 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:41.631510 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:41.639848 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:41.639885 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:41.639927 21772 server_base.cc:1034] running on GCE node
W20260927 00:45:41.639833 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:41.640576 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:41.641619 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:41.642809 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790469941642800 us; error 29 us; skew 500 ppm
Sep 27 00:45:41 dist-test-slave-pkws krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:41.645689 21772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:41.646512 21772 webserver.cc:460] Webserver started at http://127.0.0.1:43945/ using document root <none> and password file <none>
I20260927 00:45:41.647054 21772 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:41.647145 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:41.647368 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:41.649369 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "cbb93ad2746c4ab1901659ebc5294f01"
format_stamp: "Formatted at 2026-09-27 00:45:41 on dist-test-slave-pkws"
server_key: "5f7aa7d1c8c22c3240aeddafc4e4e693"
server_key_iv: "75e6374e18eddb5108317949817c528b"
server_key_version: "encryptionkey@0"
I20260927 00:45:41.649921 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbb93ad2746c4ab1901659ebc5294f01"
format_stamp: "Formatted at 2026-09-27 00:45:41 on dist-test-slave-pkws"
server_key: "5f7aa7d1c8c22c3240aeddafc4e4e693"
server_key_iv: "75e6374e18eddb5108317949817c528b"
server_key_version: "encryptionkey@0"
I20260927 00:45:41.653636 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:41.655896 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:41.657186 21772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:41.657316 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "cbb93ad2746c4ab1901659ebc5294f01"
format_stamp: "Formatted at 2026-09-27 00:45:41 on dist-test-slave-pkws"
server_key: "5f7aa7d1c8c22c3240aeddafc4e4e693"
server_key_iv: "75e6374e18eddb5108317949817c528b"
server_key_version: "encryptionkey@0"
I20260927 00:45:41.657423 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:41.684749 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:41.688081 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:41.688285 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:41.695286 21772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41765
I20260927 00:45:41.695314 21839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41765 every 8 connection(s)
I20260927 00:45:41.696269 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:41.698984 21840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:41.703681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21772
I20260927 00:45:41.703827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:41.704133 392 external_mini_cluster.cc:1428] Setting key 75508dfbe2e806186a84f785eececcb9
I20260927 00:45:41.704465 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01: Bootstrap starting.
I20260927 00:45:41.706944 21840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:41.707631 21840 log.cc:824] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:41.709429 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01: No bootstrap required, opened a new log
I20260927 00:45:41.713410 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:41.705543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:53937 (local address 127.0.0.1:41765)
0927 00:45:41.705985 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:41.705999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:41.706032 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:41.706633 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:41.706648 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:41.706983 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:41.707355 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:41.707793 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:41.708741 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:41.709694 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:41.710167 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:41.710341 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:41.711234 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:41.711240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:41.711254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:41.711791 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:41.711866 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:41.711904 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:41.712080 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:41.712493 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 00:45:41.712633 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260927 00:45:41.713515 21840 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260927 00:45:41.713773 21840 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:41.713855 21840 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb93ad2746c4ab1901659ebc5294f01, State: Initialized, Role: FOLLOWER
I20260927 00:45:41.714291 21840 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260927 00:45:41.714457 21840 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:41.714520 21840 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:41.714609 21840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:41.715603 21840 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260927 00:45:41.715973 21840 leader_election.cc:302] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbb93ad2746c4ab1901659ebc5294f01; no voters:
I20260927 00:45:41.716310 21840 leader_election.cc:288] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:41.716344 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:41.716370 21845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:41.716606 21845 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [term 1 LEADER]: Becoming Leader. State: Replica: cbb93ad2746c4ab1901659ebc5294f01, State: Running, Role: LEADER
I20260927 00:45:41.716964 21845 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } }
I20260927 00:45:41.717517 21840 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:41.718139 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbb93ad2746c4ab1901659ebc5294f01. Latest consensus state: current_term: 1 leader_uuid: "cbb93ad2746c4ab1901659ebc5294f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } } }
I20260927 00:45:41.718333 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:41.718551 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbb93ad2746c4ab1901659ebc5294f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbb93ad2746c4ab1901659ebc5294f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41765 } } }
I20260927 00:45:41.718667 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:41.718744 21857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:41.722002 21857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:41.723573 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:41.719082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:52133 (local address 127.0.0.1:41765)
0927 00:45:41.719293 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:41.719300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:41.719316 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:45:41.719447 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:41.719450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:41.719491 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:41.719540 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:41.719927 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:41.720653 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:41.721634 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:41.721943 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:41.722054 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:41.723086 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:41.723090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:41.723092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:41.723105 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:41.723133 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:41.723146 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:41.723242 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:41.723408 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 00:45:41.723441 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 00:45:41.727644 21857 catalog_manager.cc:1347] Generated new cluster ID: 4ff9ab50883e4cdfb5ddecbc06909378
I20260927 00:45:41.727702 21857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:41.739378 21857 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:41.740499 21857 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:41.753356 21857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbb93ad2746c4ab1901659ebc5294f01: Generated new TSK 0
I20260927 00:45:41.754009 21857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:41.761627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21772
W20260927 00:45:41.766830 21757 connection.cc:535] client connection to 127.0.0.1:41765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41765): Connection reset by peer (error 104)
2026-09-27T00:45:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:41 dist-test-slave-pkws krb5kdc[21871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:41 dist-test-slave-pkws krb5kdc[21871](info): set up 2 sockets
Sep 27 00:45:41 dist-test-slave-pkws krb5kdc[21871](info): commencing operation
krb5kdc: starting...
W20260927 00:45:42.821179 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:42 dist-test-slave-pkws krb5kdc[21871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 00:45:42.939433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:43.046293 21884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:43.046525 21884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:43.046592 21884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:43.050410 21884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:43.050459 21884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:43.050478 21884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:43.050534 21884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:43.050566 21884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:43.055760 21884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:43.056715 21884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:43.057725 21884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:43.065551 21890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:43.065541 21889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:43.065558 21892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:43.065932 21884 server_base.cc:1034] running on GCE node
I20260927 00:45:43.066550 21884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:43.067548 21884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:43.068744 21884 hybrid_clock.cc:648] HybridClock initialized: now 1790469943068727 us; error 41 us; skew 500 ppm
Sep 27 00:45:43 dist-test-slave-pkws krb5kdc[21871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:43.076188 21884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 00:45:43.077059 21884 webserver.cc:460] Webserver started at http://127.0.0.1:33835/ using document root <none> and password file <none>
I20260927 00:45:43.077549 21884 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:43.077605 21884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:43.077759 21884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:43.079653 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "22838de4a3024353b5b445688094420d"
format_stamp: "Formatted at 2026-09-27 00:45:43 on dist-test-slave-pkws"
server_key: "c14cfcb61fd35ceffa1181769d72d93d"
server_key_iv: "e43b6aa8b6534bdae421fe5e24f69bb0"
server_key_version: "encryptionkey@0"
I20260927 00:45:43.080153 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "22838de4a3024353b5b445688094420d"
format_stamp: "Formatted at 2026-09-27 00:45:43 on dist-test-slave-pkws"
server_key: "c14cfcb61fd35ceffa1181769d72d93d"
server_key_iv: "e43b6aa8b6534bdae421fe5e24f69bb0"
server_key_version: "encryptionkey@0"
I20260927 00:45:43.083714 21884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 00:45:43.086149 21899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:43.087291 21884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:43.087420 21884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "22838de4a3024353b5b445688094420d"
format_stamp: "Formatted at 2026-09-27 00:45:43 on dist-test-slave-pkws"
server_key: "c14cfcb61fd35ceffa1181769d72d93d"
server_key_iv: "e43b6aa8b6534bdae421fe5e24f69bb0"
server_key_version: "encryptionkey@0"
I20260927 00:45:43.087519 21884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:43.092027 21884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:43.095091 21884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:43.095275 21884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:43.101833 21884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46657
I20260927 00:45:43.101917 21951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46657 every 8 connection(s)
I20260927 00:45:43.102705 21884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:43.104799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21884
I20260927 00:45:43.104981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:43.105319 392 external_mini_cluster.cc:1428] Setting key eb66d69c35f976c5d03bab5cb758f317
I20260927 00:45:43.105625 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:43.110091 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:43.106591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:41539 (local address 127.0.0.1:46657)
0927 00:45:43.106952 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:43.106971 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:43.107004 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:43.107631 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:43.107647 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:43.107903 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:43.108167 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:43.108936 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:43.108946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:43.108966 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:43.108976 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:43.109024 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:43.109043 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:43.109146 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:43.109255 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 00:45:43.109439 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260927 00:45:43.112097 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d: Bootstrap starting.
I20260927 00:45:43.112655 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:43.114813 21952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:43.115775 21952 log.cc:824] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:43.116528 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:43.114979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.99:53621 (local address 127.0.0.1:46657)
0927 00:45:43.115104 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:43.115107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:43.115254 (+ 147us) server_negotiation.cc:407] Connection header received
0927 00:45:43.115363 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:43.115364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:43.115389 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:43.115452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:43.116102 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:43.116105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:43.116107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:43.116116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:43.116139 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:43.116148 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:43.116282 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:43.116371 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 00:45:43.116410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:43.117890 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d: No bootstrap required, opened a new log
I20260927 00:45:43.121341 21952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260927 00:45:43.121541 21952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:43.121582 21952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22838de4a3024353b5b445688094420d, State: Initialized, Role: FOLLOWER
I20260927 00:45:43.121963 21952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260927 00:45:43.122076 21952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:43.122118 21952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:43.122186 21952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:43.122956 21952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260927 00:45:43.123234 21952 leader_election.cc:302] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22838de4a3024353b5b445688094420d; no voters:
I20260927 00:45:43.123467 21952 leader_election.cc:288] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:43.123589 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:43.123827 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [term 1 LEADER]: Becoming Leader. State: Replica: 22838de4a3024353b5b445688094420d, State: Running, Role: LEADER
I20260927 00:45:43.124156 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } }
I20260927 00:45:43.124675 21952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:43.125829 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22838de4a3024353b5b445688094420d. Latest consensus state: current_term: 1 leader_uuid: "22838de4a3024353b5b445688094420d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } } }
I20260927 00:45:43.125962 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:43.126325 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22838de4a3024353b5b445688094420d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22838de4a3024353b5b445688094420d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46657 } } }
I20260927 00:45:43.126513 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:43.126442 21971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:43.129848 21971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:43.135479 21971 catalog_manager.cc:1347] Generated new cluster ID: 3f59735e660b49919b92f6a0c649bdf9
I20260927 00:45:43.135541 21971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:43.145653 21971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:43.145954 21971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:43.152709 21971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22838de4a3024353b5b445688094420d: Generated new TSK 0
I20260927 00:45:43.153475 21971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:43.161818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21884
2026-09-27T00:45:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:43 dist-test-slave-pkws krb5kdc[21983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:43 dist-test-slave-pkws krb5kdc[21983](info): set up 2 sockets
Sep 27 00:45:43 dist-test-slave-pkws krb5kdc[21983](info): commencing operation
krb5kdc: starting...
W20260927 00:45:44.197089 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:44.315727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:44.421834 21996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:44.422075 21996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:44.422147 21996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:44.426095 21996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:44.426147 21996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:44.426188 21996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:44.426227 21996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:44.426267 21996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:44.431875 21996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43949
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:44.432843 21996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:44.433817 21996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:44.441697 22002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:44.441705 22004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:44.441712 22001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:44.442056 21996 server_base.cc:1034] running on GCE node
I20260927 00:45:44.442585 21996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:44.443605 21996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:44.444792 21996 hybrid_clock.cc:648] HybridClock initialized: now 1790469944444778 us; error 35 us; skew 500 ppm
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:44.447438 21996 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:44.448235 21996 webserver.cc:460] Webserver started at http://127.0.98.62:45951/ using document root <none> and password file <none>
I20260927 00:45:44.448766 21996 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:44.448856 21996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:44.449103 21996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:44.451095 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "d090a648d0d548f4b955ddf5ffc1af27"
format_stamp: "Formatted at 2026-09-27 00:45:44 on dist-test-slave-pkws"
server_key: "860d0ecee28c33196ced289b92b13fce"
server_key_iv: "18fb079a689380b5f148cc2ccfad6618"
server_key_version: "encryptionkey@0"
I20260927 00:45:44.451650 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "d090a648d0d548f4b955ddf5ffc1af27"
format_stamp: "Formatted at 2026-09-27 00:45:44 on dist-test-slave-pkws"
server_key: "860d0ecee28c33196ced289b92b13fce"
server_key_iv: "18fb079a689380b5f148cc2ccfad6618"
server_key_version: "encryptionkey@0"
I20260927 00:45:44.455138 21996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 00:45:44.457485 22011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:44.458626 21996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 00:45:44.458765 21996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "d090a648d0d548f4b955ddf5ffc1af27"
format_stamp: "Formatted at 2026-09-27 00:45:44 on dist-test-slave-pkws"
server_key: "860d0ecee28c33196ced289b92b13fce"
server_key_iv: "18fb079a689380b5f148cc2ccfad6618"
server_key_version: "encryptionkey@0"
I20260927 00:45:44.458871 21996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:44.480064 21996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:44.482720 21996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:44.482910 21996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:44.489547 21996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43949
I20260927 00:45:44.489573 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43949 every 8 connection(s)
I20260927 00:45:44.490397 21996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:44.490986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 21996
I20260927 00:45:44.491138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:44.491421 392 external_mini_cluster.cc:1428] Setting key ac2724e4c8a6193346c702b1b89b15e4
I20260927 00:45:44.493397 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:44.499627 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27: Bootstrap starting.
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[21983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:44.502146 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:44.503036 22064 log.cc:824] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:44.505242 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27: No bootstrap required, opened a new log
I20260927 00:45:44.506595 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:44.492754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42645 (local address 127.0.98.62:43949)
0927 00:45:44.493196 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:44.493214 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:44.493253 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:45:44.494067 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:44.494087 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:44.494473 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:44.494821 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:44.495467 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:44.496966 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:44.497909 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:44.498588 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:44.498748 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:44.501478 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:44.501484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:44.501499 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:44.501527 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:44.503435 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:44.503998 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:44.504004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:44.504009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:44.504129 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:44.504426 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:44.504429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:44.504430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:44.504910 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:44.505107 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:44.505356 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 00:45:44.505742 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":159,"threads_started":1}
I20260927 00:45:44.509470 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:44.509366 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260927 00:45:44.509609 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:44.509684 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d090a648d0d548f4b955ddf5ffc1af27, State: Initialized, Role: FOLLOWER
I20260927 00:45:44.510111 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260927 00:45:44.510253 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:44.510326 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:44.510421 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:44.511278 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260927 00:45:44.511634 22064 leader_election.cc:302] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d090a648d0d548f4b955ddf5ffc1af27; no voters:
I20260927 00:45:44.511955 22064 leader_election.cc:288] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:44.512018 22074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:44.512260 22074 raft_consensus.cc:686] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [term 1 LEADER]: Becoming Leader. State: Replica: d090a648d0d548f4b955ddf5ffc1af27, State: Running, Role: LEADER
I20260927 00:45:44.512575 22074 consensus_queue.cc:237] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } }
I20260927 00:45:44.513208 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:44.513728 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d090a648d0d548f4b955ddf5ffc1af27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } } }
I20260927 00:45:44.513906 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:44.514113 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d090a648d0d548f4b955ddf5ffc1af27. Latest consensus state: current_term: 1 leader_uuid: "d090a648d0d548f4b955ddf5ffc1af27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d090a648d0d548f4b955ddf5ffc1af27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43949 } } }
I20260927 00:45:44.514273 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:44.514318 22082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:44.517444 22082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:44.520596 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:44.511464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57623 (local address 127.0.98.62:43949)
0927 00:45:44.511562 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:44.511567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:44.511785 (+ 218us) server_negotiation.cc:407] Connection header received
0927 00:45:44.511896 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:44.511901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:44.511946 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:44.512032 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:44.512781 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:44.513436 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:44.514601 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:44.514923 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:44.515023 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:44.517180 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:44.517184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:44.517186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:44.517209 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:44.518865 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:44.519336 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:44.519339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:44.519341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:44.519422 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:44.519764 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:44.519767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:44.519768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:44.519933 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:44.520082 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:44.520344 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 00:45:44.520432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 00:45:44.522763 22082 catalog_manager.cc:1347] Generated new cluster ID: 066ad3c0f85442958b13b4e92508fe8b
I20260927 00:45:44.522868 22082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:44.531738 22082 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:44.532825 22082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:44.539628 22082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d090a648d0d548f4b955ddf5ffc1af27: Generated new TSK 0
I20260927 00:45:44.540226 22082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:44.548202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 21996
W20260927 00:45:44.553524 21981 connection.cc:535] client connection to 127.0.98.62:43949 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43949): Connection reset by peer (error 104)
2026-09-27T00:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[22095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[22095](info): set up 2 sockets
Sep 27 00:45:44 dist-test-slave-pkws krb5kdc[22095](info): commencing operation
krb5kdc: starting...
W20260927 00:45:45.585016 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:45 dist-test-slave-pkws krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:45.704560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:45.810662 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:45.810912 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:45.810953 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:45.814786 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:45.814832 22108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:45.814852 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:45.814881 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:45.814896 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:45.819962 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:45.820854 22108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:45.821798 22108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:45.830966 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:45.831135 22108 server_base.cc:1034] running on GCE node
W20260927 00:45:45.830966 22114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:45.831329 22116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:45.832002 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:45.833122 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:45.834306 22108 hybrid_clock.cc:648] HybridClock initialized: now 1790469945834278 us; error 54 us; skew 500 ppm
Sep 27 00:45:45 dist-test-slave-pkws krb5kdc[22095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:45.836982 22108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:45.837847 22108 webserver.cc:460] Webserver started at http://127.0.98.62:39259/ using document root <none> and password file <none>
I20260927 00:45:45.838384 22108 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:45.838491 22108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:45.838712 22108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:45.840704 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "16822a50d0a1455195848cd2e05f853d"
format_stamp: "Formatted at 2026-09-27 00:45:45 on dist-test-slave-pkws"
server_key: "9a812385f159f946974b68f6a77155cf"
server_key_iv: "f06e712231e7421e9160b07df35ef398"
server_key_version: "encryptionkey@0"
I20260927 00:45:45.841332 22108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "16822a50d0a1455195848cd2e05f853d"
format_stamp: "Formatted at 2026-09-27 00:45:45 on dist-test-slave-pkws"
server_key: "9a812385f159f946974b68f6a77155cf"
server_key_iv: "f06e712231e7421e9160b07df35ef398"
server_key_version: "encryptionkey@0"
I20260927 00:45:45.844794 22108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 00:45:45.847092 22123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:45.848248 22108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:45.848393 22108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "16822a50d0a1455195848cd2e05f853d"
format_stamp: "Formatted at 2026-09-27 00:45:45 on dist-test-slave-pkws"
server_key: "9a812385f159f946974b68f6a77155cf"
server_key_iv: "f06e712231e7421e9160b07df35ef398"
server_key_version: "encryptionkey@0"
I20260927 00:45:45.848510 22108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:45.861019 22108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:45.863659 22108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:45.863857 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:45.871052 22108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43783
I20260927 00:45:45.871068 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43783 every 8 connection(s)
I20260927 00:45:45.871953 22108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:45.874667 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:45.879787 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: Bootstrap starting.
I20260927 00:45:45.880008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22108
I20260927 00:45:45.880158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:45.880429 392 external_mini_cluster.cc:1428] Setting key b0ab09afdb73d36cbd6142dc8d5b7fe5
I20260927 00:45:45.882267 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:45.882939 22176 log.cc:824] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:45.884778 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: No bootstrap required, opened a new log
I20260927 00:45:45.888185 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } }
I20260927 00:45:45.888468 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:45.888549 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16822a50d0a1455195848cd2e05f853d, State: Initialized, Role: FOLLOWER
I20260927 00:45:45.889086 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } }
I20260927 00:45:45.889196 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:45.881680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40017 (local address 127.0.98.62:43783)
0927 00:45:45.882065 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:45.882082 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:45.882115 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:45.882738 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:45.882755 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:45.883138 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:45.883516 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:45.883896 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:45.884817 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:45.885769 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:45.886238 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:45.886412 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:45.887339 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:45.887344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:45.887356 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:45.887734 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:45.887784 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:45.887805 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:45.887947 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:45.888127 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 00:45:45.888258 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260927 00:45:45.889235 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:45.889330 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:45.889442 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:45.890400 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } }
I20260927 00:45:45.890725 22176 leader_election.cc:302] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16822a50d0a1455195848cd2e05f853d; no voters:
I20260927 00:45:45.891057 22176 leader_election.cc:288] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:45.891144 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:45.891412 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [term 1 LEADER]: Becoming Leader. State: Replica: 16822a50d0a1455195848cd2e05f853d, State: Running, Role: LEADER
I20260927 00:45:45.891772 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } }
I20260927 00:45:45.892180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:45.892391 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:45.893684 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16822a50d0a1455195848cd2e05f853d. Latest consensus state: current_term: 1 leader_uuid: "16822a50d0a1455195848cd2e05f853d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } }
I20260927 00:45:45.893687 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16822a50d0a1455195848cd2e05f853d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16822a50d0a1455195848cd2e05f853d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } }
I20260927 00:45:45.893898 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:45.894186 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d [sys.catalog]: This master's current role is: LEADER
W20260927 00:45:45.897199 22200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:45.897284 22200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:45.897368 22197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:45.898051 22197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:45.899801 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:45.895247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43979 (local address 127.0.98.62:43783)
0927 00:45:45.895339 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:45.895343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:45.895639 (+ 296us) server_negotiation.cc:407] Connection header received
0927 00:45:45.895737 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:45.895740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:45.895779 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:45.895858 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:45.896141 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:45.896939 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:45.898067 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:45.898367 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:45.898472 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:45.899330 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:45.899334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:45.899337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:45.899349 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:45.899376 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:45.899388 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:45.899475 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:45.899651 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 00:45:45.899682 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:45:45.903232 22197 catalog_manager.cc:1347] Generated new cluster ID: 2dced376e4d04abd847e7a7008a13b63
I20260927 00:45:45.903291 22197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:45.921458 22197 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:45.922344 22197 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:45.930341 22197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16822a50d0a1455195848cd2e05f853d: Generated new TSK 0
I20260927 00:45:45.931156 22197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:45.941234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22108
W20260927 00:45:45.946425 22093 connection.cc:535] client connection to 127.0.98.62:43783 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43783): Connection reset by peer (error 104)
2026-09-27T00:45:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:45 dist-test-slave-pkws krb5kdc[22207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:45 dist-test-slave-pkws krb5kdc[22207](info): set up 2 sockets
Sep 27 00:45:45 dist-test-slave-pkws krb5kdc[22207](info): commencing operation
krb5kdc: starting...
W20260927 00:45:46.989125 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:47 dist-test-slave-pkws krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:47.107514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:47.214040 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:47.214291 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:47.214359 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:47.218387 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:47.218446 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:47.218465 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:47.218524 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:47.218557 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:47.223695 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:47.224622 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:47.225613 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:47.233562 22220 server_base.cc:1034] running on GCE node
W20260927 00:45:47.233532 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:47.233500 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:47.233507 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:47.234236 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:47.235210 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:47.236394 22220 hybrid_clock.cc:648] HybridClock initialized: now 1790469947236386 us; error 28 us; skew 500 ppm
Sep 27 00:45:47 dist-test-slave-pkws krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:47.239101 22220 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:47.239931 22220 webserver.cc:460] Webserver started at http://127.0.98.62:35997/ using document root <none> and password file <none>
I20260927 00:45:47.240455 22220 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:47.240543 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:47.240762 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:47.242748 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0"
format_stamp: "Formatted at 2026-09-27 00:45:47 on dist-test-slave-pkws"
server_key: "81e48f48fb27bccce3bd5c9aca48653d"
server_key_iv: "a53e1667bd4591e0e830377b53a69022"
server_key_version: "encryptionkey@0"
I20260927 00:45:47.243287 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0"
format_stamp: "Formatted at 2026-09-27 00:45:47 on dist-test-slave-pkws"
server_key: "81e48f48fb27bccce3bd5c9aca48653d"
server_key_iv: "a53e1667bd4591e0e830377b53a69022"
server_key_version: "encryptionkey@0"
I20260927 00:45:47.246760 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:47.249024 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:47.250106 22220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:47.250239 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0"
format_stamp: "Formatted at 2026-09-27 00:45:47 on dist-test-slave-pkws"
server_key: "81e48f48fb27bccce3bd5c9aca48653d"
server_key_iv: "a53e1667bd4591e0e830377b53a69022"
server_key_version: "encryptionkey@0"
I20260927 00:45:47.250344 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:47.254822 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:47.257494 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:47.257712 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:47.264523 22220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34455
I20260927 00:45:47.264550 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34455 every 8 connection(s)
I20260927 00:45:47.265367 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:47.267773 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:47.272941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22220
I20260927 00:45:47.273043 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0: Bootstrap starting.
I20260927 00:45:47.273089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:47.273348 392 external_mini_cluster.cc:1428] Setting key abcea562d10d96e6c99776b0e0624f17
I20260927 00:45:47.275550 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:47.276396 22288 log.cc:824] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:47.278429 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0: No bootstrap required, opened a new log
I20260927 00:45:47.278627 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:47.274591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37281 (local address 127.0.98.62:34455)
0927 00:45:47.275002 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:47.275018 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:47.275059 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:45:47.275941 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:47.275971 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:47.276321 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:47.276656 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:47.277479 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:47.277483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:47.277494 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:47.277502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:47.277545 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:47.277562 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:47.277642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:47.277772 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 00:45:47.277934 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":91,"threads_started":1}
I20260927 00:45:47.281169 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:47.282260 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260927 00:45:47.282527 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:47.282598 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80a1fb7a1824ddeaa19a0ecec7c44f0, State: Initialized, Role: FOLLOWER
I20260927 00:45:47.283075 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260927 00:45:47.283269 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:47.283376 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:47.283489 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:47.284621 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260927 00:45:47.284788 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:47.283289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48965 (local address 127.0.98.62:34455)
0927 00:45:47.283397 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:47.283401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:47.283531 (+ 130us) server_negotiation.cc:407] Connection header received
0927 00:45:47.283667 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:47.283669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:47.283695 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:47.283754 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:47.284391 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:47.284394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:47.284395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:47.284405 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:47.284425 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:47.284434 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:47.284501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:47.284621 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 00:45:47.284649 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:45:47.285064 22288 leader_election.cc:302] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80a1fb7a1824ddeaa19a0ecec7c44f0; no voters:
I20260927 00:45:47.285386 22288 leader_election.cc:288] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:47.285478 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:47.285707 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [term 1 LEADER]: Becoming Leader. State: Replica: b80a1fb7a1824ddeaa19a0ecec7c44f0, State: Running, Role: LEADER
I20260927 00:45:47.286015 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260927 00:45:47.286391 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:47.287151 22298 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80a1fb7a1824ddeaa19a0ecec7c44f0. Latest consensus state: current_term: 1 leader_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260927 00:45:47.287336 22298 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:47.287559 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80a1fb7a1824ddeaa19a0ecec7c44f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260927 00:45:47.287690 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:47.287777 22307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:47.290994 22307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:47.296192 22307 catalog_manager.cc:1347] Generated new cluster ID: d7f23cf156164b99b220ccbb545b9ece
I20260927 00:45:47.296270 22307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:47.311857 22307 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:47.312165 22307 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:47.317138 22307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80a1fb7a1824ddeaa19a0ecec7c44f0: Generated new TSK 0
I20260927 00:45:47.317986 22307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:47.330246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22220
2026-09-27T00:45:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:47 dist-test-slave-pkws krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:47 dist-test-slave-pkws krb5kdc[22319](info): set up 2 sockets
Sep 27 00:45:47 dist-test-slave-pkws krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260927 00:45:48.365175 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:48.483737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:48.589704 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:48.589923 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:48.589962 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:48.593860 22332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:48.593910 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:48.593930 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:48.593945 22332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:48.593981 22332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:48.599090 22332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:48.600023 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:48.601004 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:48.609201 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:48.609207 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:48.609210 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:48.609493 22332 server_base.cc:1034] running on GCE node
I20260927 00:45:48.610033 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:48.611089 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:48.612248 22332 hybrid_clock.cc:648] HybridClock initialized: now 1790469948612236 us; error 23 us; skew 500 ppm
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:48.620333 22332 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:48.621229 22332 webserver.cc:460] Webserver started at http://127.0.98.62:34961/ using document root <none> and password file <none>
I20260927 00:45:48.621769 22332 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:48.621862 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:48.622100 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:48.624058 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "3a927fd20db44d23a58b942a966501f3"
format_stamp: "Formatted at 2026-09-27 00:45:48 on dist-test-slave-pkws"
server_key: "828a19da084a9229b8d739a54a251c02"
server_key_iv: "3944e79a47cd9dbb57c4f6eff7f4fc40"
server_key_version: "encryptionkey@0"
I20260927 00:45:48.624593 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a927fd20db44d23a58b942a966501f3"
format_stamp: "Formatted at 2026-09-27 00:45:48 on dist-test-slave-pkws"
server_key: "828a19da084a9229b8d739a54a251c02"
server_key_iv: "3944e79a47cd9dbb57c4f6eff7f4fc40"
server_key_version: "encryptionkey@0"
I20260927 00:45:48.628120 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:48.630470 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:48.631599 22332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:48.631754 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "3a927fd20db44d23a58b942a966501f3"
format_stamp: "Formatted at 2026-09-27 00:45:48 on dist-test-slave-pkws"
server_key: "828a19da084a9229b8d739a54a251c02"
server_key_iv: "3944e79a47cd9dbb57c4f6eff7f4fc40"
server_key_version: "encryptionkey@0"
I20260927 00:45:48.631860 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:48.647832 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:48.650704 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:48.650899 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:48.657956 22332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38115
I20260927 00:45:48.657982 22399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38115 every 8 connection(s)
I20260927 00:45:48.658862 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:48.661382 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:48.666451 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: Bootstrap starting.
I20260927 00:45:48.668673 22400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:48.669150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22332
I20260927 00:45:48.669282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:48.669610 392 external_mini_cluster.cc:1428] Setting key a8a033f02260b80392fd138f600f3628
I20260927 00:45:48.669591 22400 log.cc:824] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:48.671556 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: No bootstrap required, opened a new log
I20260927 00:45:48.675532 22400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } }
I20260927 00:45:48.675753 22400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:48.675846 22400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a927fd20db44d23a58b942a966501f3, State: Initialized, Role: FOLLOWER
I20260927 00:45:48.676327 22400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } }
I20260927 00:45:48.676491 22400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:48.676564 22400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:48.676664 22400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:48.677520 22400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } }
I20260927 00:45:48.677866 22400 leader_election.cc:302] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a927fd20db44d23a58b942a966501f3; no voters:
I20260927 00:45:48.678140 22400 leader_election.cc:288] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:48.678246 22405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:48.678469 22405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a927fd20db44d23a58b942a966501f3, State: Running, Role: LEADER
I20260927 00:45:48.678843 22405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } }
I20260927 00:45:48.679366 22400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:48.680609 22406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a927fd20db44d23a58b942a966501f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } } }
I20260927 00:45:48.680728 22406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:48.681100 22407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a927fd20db44d23a58b942a966501f3. Latest consensus state: current_term: 1 leader_uuid: "3a927fd20db44d23a58b942a966501f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a927fd20db44d23a58b942a966501f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38115 } } }
I20260927 00:45:48.681246 22407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:48.684015 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:48.670701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59303 (local address 127.0.98.62:38115)
0927 00:45:48.671049 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:48.671073 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:48.671109 (+ 36us) server_negotiation.cc:407] Connection header received
0927 00:45:48.671766 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:48.671783 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:48.672099 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:48.672363 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:48.673240 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:48.674165 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:48.675335 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:48.675838 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:48.676032 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:48.678825 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:48.678831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:48.678846 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:48.678877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:48.681068 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:48.681611 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:48.681620 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:48.681629 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:48.681760 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:48.682130 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:48.682135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:48.682137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:48.682512 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:48.682727 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:48.683006 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 00:45:48.683330 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
W20260927 00:45:48.684284 22420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:48.684370 22420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:48.684427 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:48.685259 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:48.686765 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:48.690868 22414 catalog_manager.cc:1347] Generated new cluster ID: 34f3d2b3b9224a85ad941af500d1aeb6
I20260927 00:45:48.690923 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:48.697136 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:48.689066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34715 (local address 127.0.98.62:38115)
0927 00:45:48.689166 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:48.689176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:48.689214 (+ 38us) server_negotiation.cc:407] Connection header received
0927 00:45:48.689403 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:48.689407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:48.689459 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:48.689555 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:48.690228 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:48.690903 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:48.692012 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:48.692391 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:48.692498 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:48.693926 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:48.693930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:48.693933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:48.693957 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:48.695519 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:48.695954 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:48.695957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:48.695959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:48.696038 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:48.696323 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:48.696326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:48.696328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:48.696524 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:48.696647 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:48.696858 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 00:45:48.696996 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:45:48.709012 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:48.709839 22414 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:48.722020 22414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a927fd20db44d23a58b942a966501f3: Generated new TSK 0
I20260927 00:45:48.722664 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:48.731328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22332
2026-09-27T00:45:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22431](info): set up 2 sockets
Sep 27 00:45:48 dist-test-slave-pkws krb5kdc[22431](info): commencing operation
krb5kdc: starting...
W20260927 00:45:49.781181 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:49 dist-test-slave-pkws krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:49.905238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:50.012240 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:50.012545 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:50.012619 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:50.016582 22444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:50.016664 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:50.016701 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:50.016754 22444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:50.016791 22444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:50.021878 22444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:50.022856 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:50.023811 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:50.031993 22450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:50.032015 22452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:50.032047 22449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:50.032015 22444 server_base.cc:1034] running on GCE node
I20260927 00:45:50.032786 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:50.033855 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:50.035012 22444 hybrid_clock.cc:648] HybridClock initialized: now 1790469950034993 us; error 30 us; skew 500 ppm
Sep 27 00:45:50 dist-test-slave-pkws krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:50.037634 22444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:50.038434 22444 webserver.cc:460] Webserver started at http://127.0.98.62:42711/ using document root <none> and password file <none>
I20260927 00:45:50.038960 22444 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:50.039018 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:50.039242 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:50.041254 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "70351411c9bf4394b423ca6a99153f4f"
format_stamp: "Formatted at 2026-09-27 00:45:50 on dist-test-slave-pkws"
server_key: "211921c052ccd06a97c463b17bb3e4e4"
server_key_iv: "52971fcc32ca484588d52e09b2f92a7e"
server_key_version: "encryptionkey@0"
I20260927 00:45:50.041786 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "70351411c9bf4394b423ca6a99153f4f"
format_stamp: "Formatted at 2026-09-27 00:45:50 on dist-test-slave-pkws"
server_key: "211921c052ccd06a97c463b17bb3e4e4"
server_key_iv: "52971fcc32ca484588d52e09b2f92a7e"
server_key_version: "encryptionkey@0"
I20260927 00:45:50.045222 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 00:45:50.047515 22459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:50.048640 22444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:50.048766 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "70351411c9bf4394b423ca6a99153f4f"
format_stamp: "Formatted at 2026-09-27 00:45:50 on dist-test-slave-pkws"
server_key: "211921c052ccd06a97c463b17bb3e4e4"
server_key_iv: "52971fcc32ca484588d52e09b2f92a7e"
server_key_version: "encryptionkey@0"
I20260927 00:45:50.048870 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:50.059453 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:50.062125 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:50.062314 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:50.068849 22444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43189
I20260927 00:45:50.068956 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43189 every 8 connection(s)
I20260927 00:45:50.069795 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:50.070302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22444
I20260927 00:45:50.070439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:50.070766 392 external_mini_cluster.cc:1428] Setting key 0b330bea78e6fa40bdee499b5199cece
I20260927 00:45:50.072834 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:50.078078 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f: Bootstrap starting.
I20260927 00:45:50.079166 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:50.072098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60863 (local address 127.0.98.62:43189)
0927 00:45:50.072501 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:50.072518 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:50.072551 (+ 33us) server_negotiation.cc:407] Connection header received
0927 00:45:50.073252 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:50.073269 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:50.073580 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:50.073929 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:50.074348 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:50.075314 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:50.076192 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:50.076668 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:50.076843 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:50.077656 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:50.077661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:50.077672 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:50.078039 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:50.078087 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:50.078109 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:50.078219 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:50.078353 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:45:50.078547 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":126,"threads_started":1}
I20260927 00:45:50.080408 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:50.081228 22512 log.cc:824] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:50.081985 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:50.083381 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f: No bootstrap required, opened a new log
I20260927 00:45:50.087831 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260927 00:45:50.088088 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:50.084062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51739 (local address 127.0.98.62:43189)
0927 00:45:50.084173 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:50.084178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:50.084436 (+ 258us) server_negotiation.cc:407] Connection header received
0927 00:45:50.084519 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:50.084521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:50.084556 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:50.084623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:50.084944 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:50.085474 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:50.086659 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:50.086872 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:50.086963 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:50.087687 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:50.087689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:50.087691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:50.087701 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:50.087721 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:50.087731 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:50.087785 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:50.087907 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 00:45:50.087947 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:50.088095 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:50.088222 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70351411c9bf4394b423ca6a99153f4f, State: Initialized, Role: FOLLOWER
I20260927 00:45:50.088662 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260927 00:45:50.088804 22512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:50.088863 22512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:50.088977 22512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:50.089761 22512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260927 00:45:50.090090 22512 leader_election.cc:302] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70351411c9bf4394b423ca6a99153f4f; no voters:
I20260927 00:45:50.090391 22512 leader_election.cc:288] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:50.090451 22522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:50.090651 22522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [term 1 LEADER]: Becoming Leader. State: Replica: 70351411c9bf4394b423ca6a99153f4f, State: Running, Role: LEADER
I20260927 00:45:50.090991 22522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } }
I20260927 00:45:50.091548 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:50.092074 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70351411c9bf4394b423ca6a99153f4f. Latest consensus state: current_term: 1 leader_uuid: "70351411c9bf4394b423ca6a99153f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } }
I20260927 00:45:50.092283 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:50.092460 22523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70351411c9bf4394b423ca6a99153f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70351411c9bf4394b423ca6a99153f4f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } }
I20260927 00:45:50.092628 22523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:50.092689 22531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:50.096035 22531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:50.101511 22531 catalog_manager.cc:1347] Generated new cluster ID: 7c0d25f6bbfb4ab3a3e3f3cd4c9a2bd0
I20260927 00:45:50.101568 22531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:50.113580 22531 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:50.114738 22531 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:50.121608 22531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70351411c9bf4394b423ca6a99153f4f: Generated new TSK 0
I20260927 00:45:50.122177 22531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:50.125617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22444
2026-09-27T00:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:50 dist-test-slave-pkws krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:50 dist-test-slave-pkws krb5kdc[22543](info): set up 2 sockets
Sep 27 00:45:50 dist-test-slave-pkws krb5kdc[22543](info): commencing operation
krb5kdc: starting...
W20260927 00:45:51.149219 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:51 dist-test-slave-pkws krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:51.265791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:51.372637 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:51.372871 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:51.372993 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:51.376929 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:51.376982 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:51.377002 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:51.377018 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:51.377033 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:51.382097 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:51.382961 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:51.383865 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:51.392222 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:51.392222 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:51.392339 22556 server_base.cc:1034] running on GCE node
W20260927 00:45:51.392258 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:51.393097 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:51.394164 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:51.395376 22556 hybrid_clock.cc:648] HybridClock initialized: now 1790469951395361 us; error 34 us; skew 500 ppm
Sep 27 00:45:51 dist-test-slave-pkws krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:51.398002 22556 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:51.398834 22556 webserver.cc:460] Webserver started at http://127.0.98.62:43837/ using document root <none> and password file <none>
I20260927 00:45:51.399384 22556 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:51.399473 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:51.399695 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:51.401772 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "0f67775850d249a69a88eadaa21fed06"
format_stamp: "Formatted at 2026-09-27 00:45:51 on dist-test-slave-pkws"
server_key: "6fbbe1289c7efa146e5cc959311e8866"
server_key_iv: "473b55f3708a8799c5459a75be1bd992"
server_key_version: "encryptionkey@0"
I20260927 00:45:51.402336 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f67775850d249a69a88eadaa21fed06"
format_stamp: "Formatted at 2026-09-27 00:45:51 on dist-test-slave-pkws"
server_key: "6fbbe1289c7efa146e5cc959311e8866"
server_key_iv: "473b55f3708a8799c5459a75be1bd992"
server_key_version: "encryptionkey@0"
I20260927 00:45:51.405861 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 00:45:51.408223 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:51.409533 22556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:51.409667 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "0f67775850d249a69a88eadaa21fed06"
format_stamp: "Formatted at 2026-09-27 00:45:51 on dist-test-slave-pkws"
server_key: "6fbbe1289c7efa146e5cc959311e8866"
server_key_iv: "473b55f3708a8799c5459a75be1bd992"
server_key_version: "encryptionkey@0"
I20260927 00:45:51.409778 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:51.414076 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:51.416779 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:51.416993 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:51.424263 22556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46379
I20260927 00:45:51.424316 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46379 every 8 connection(s)
I20260927 00:45:51.425283 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:51.427834 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:51.431756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22556
I20260927 00:45:51.431903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:51.432180 392 external_mini_cluster.cc:1428] Setting key 4591cb02b654d03e4476e3731b34a24c
I20260927 00:45:51.433416 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06: Bootstrap starting.
I20260927 00:45:51.436038 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:51.436846 22624 log.cc:824] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:51.437520 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:51.433550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50871 (local address 127.0.98.62:46379)
0927 00:45:51.433955 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:51.433972 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:51.434013 (+ 41us) server_negotiation.cc:407] Connection header received
0927 00:45:51.434846 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:51.434868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:51.435195 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:51.435544 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:51.436255 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:51.436261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:51.436272 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:51.436283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:51.436353 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:51.436373 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:51.436484 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:51.436635 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 00:45:51.436781 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260927 00:45:51.438793 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06: No bootstrap required, opened a new log
I20260927 00:45:51.440151 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:51.442755 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } }
I20260927 00:45:51.443033 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 00:45:51.443100 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f67775850d249a69a88eadaa21fed06, State: Initialized, Role: FOLLOWER
I20260927 00:45:51.443548 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } }
I20260927 00:45:51.443672 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:51.442082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34121 (local address 127.0.98.62:46379)
0927 00:45:51.442184 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:51.442190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:51.442362 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:45:51.442508 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:51.442513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:51.442546 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:51.442625 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:51.443318 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:51.443321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:51.443323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 00:45:51.443335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:51.443359 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 00:45:51.443375 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:51.443461 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:51.443532 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 00:45:51.443567 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 00:45:51.443724 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 00:45:51.443863 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 00:45:51.444002 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:51.444996 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } }
I20260927 00:45:51.445359 22624 leader_election.cc:302] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f67775850d249a69a88eadaa21fed06; no voters:
I20260927 00:45:51.445642 22624 leader_election.cc:288] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 00:45:51.445739 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:51.445976 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [term 1 LEADER]: Becoming Leader. State: Replica: 0f67775850d249a69a88eadaa21fed06, State: Running, Role: LEADER
I20260927 00:45:51.446286 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } }
I20260927 00:45:51.446794 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:51.447870 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f67775850d249a69a88eadaa21fed06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } } }
I20260927 00:45:51.447873 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f67775850d249a69a88eadaa21fed06. Latest consensus state: current_term: 1 leader_uuid: "0f67775850d249a69a88eadaa21fed06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f67775850d249a69a88eadaa21fed06" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46379 } } }
I20260927 00:45:51.448045 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:51.448058 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06 [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:51.448421 22643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:51.451468 22643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:51.457265 22643 catalog_manager.cc:1347] Generated new cluster ID: a05fbde2e8db48099c120b73430b851d
I20260927 00:45:51.457365 22643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:51.474534 22643 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:51.474843 22643 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:51.486574 22643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f67775850d249a69a88eadaa21fed06: Generated new TSK 0
I20260927 00:45:51.487308 22643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 00:45:51.496196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22556
2026-09-27T00:45:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (25104 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:51 dist-test-slave-pkws krb5kdc[22655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:51 dist-test-slave-pkws krb5kdc[22655](info): set up 2 sockets
Sep 27 00:45:51 dist-test-slave-pkws krb5kdc[22655](info): commencing operation
krb5kdc: starting...
W20260927 00:45:52.589177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:52 dist-test-slave-pkws krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:52.703814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:52.810599 22668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:52.810891 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:52.810942 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:52.811010 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:52.814904 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:52.814987 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:52.815026 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:52.815069 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:52.815104 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:52.820152 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:52.821108 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:52.822055 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:52.830232 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:52.830358 22668 server_base.cc:1034] running on GCE node
W20260927 00:45:52.830240 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:52.830240 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:52.831037 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:52.832051 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:52.833227 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790469952833202 us; error 39 us; skew 500 ppm
Sep 27 00:45:52 dist-test-slave-pkws krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:52.835714 22668 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:52.836508 22668 webserver.cc:460] Webserver started at http://127.0.98.62:36531/ using document root <none> and password file <none>
I20260927 00:45:52.837091 22668 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:52.837176 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:52.837395 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:52.839289 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "33e8f6c719fc4bf487d82abd9de9587a"
format_stamp: "Formatted at 2026-09-27 00:45:52 on dist-test-slave-pkws"
server_key: "0344c9e5f5a54494f306858452ab32bf"
server_key_iv: "e72ab2b379fef13546ed4286c1eb0e67"
server_key_version: "encryptionkey@0"
I20260927 00:45:52.839838 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "33e8f6c719fc4bf487d82abd9de9587a"
format_stamp: "Formatted at 2026-09-27 00:45:52 on dist-test-slave-pkws"
server_key: "0344c9e5f5a54494f306858452ab32bf"
server_key_iv: "e72ab2b379fef13546ed4286c1eb0e67"
server_key_version: "encryptionkey@0"
I20260927 00:45:52.843312 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:52.845562 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:52.846674 22668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:52.846845 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "33e8f6c719fc4bf487d82abd9de9587a"
format_stamp: "Formatted at 2026-09-27 00:45:52 on dist-test-slave-pkws"
server_key: "0344c9e5f5a54494f306858452ab32bf"
server_key_iv: "e72ab2b379fef13546ed4286c1eb0e67"
server_key_version: "encryptionkey@0"
I20260927 00:45:52.846951 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:52.860294 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:52.862838 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:52.863026 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:52.869493 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36299
I20260927 00:45:52.869791 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36299 every 8 connection(s)
I20260927 00:45:52.870388 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:52.872982 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:52.876638 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:52.878885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22668
I20260927 00:45:52.879031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:52.879298 392 external_mini_cluster.cc:1428] Setting key 296ee3cfdf8f6ebed92cafae78811895
Sep 27 00:45:52 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 00:45:52.890951 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.0.98.62:36299)
0927 00:45:52.879064 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:52.879082 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:52.879432 (+ 350us) server_negotiation.cc:407] Connection header received
0927 00:45:52.880437 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:52.880441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:52.880605 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:52.880780 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:52.881572 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:52.882421 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:52.883407 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:52.883765 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:52.883930 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:52.886317 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:52.886325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:52.886340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:52.886367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:52.887951 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:52.888438 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:52.888443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:52.888445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:52.888548 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:52.888871 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:52.888877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:52.888879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:52.889241 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:52.889377 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:52.889738 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 00:45:52.889983 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":252,"thread_start_us":125,"threads_started":1}
I20260927 00:45:52.890951 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.878684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.0.1:46680)
0927 00:45:52.879286 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:52.879321 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:52.880148 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:52.880777 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:52.880782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:52.881211 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:52.881424 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:52.881435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:52.882563 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:52.882566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:52.883286 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:52.883294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:52.883911 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:52.883914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:52.884124 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:52.884687 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:52.884702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:52.886168 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:52.888088 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:52.888097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:52.888106 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:52.888327 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:52.888646 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:52.888651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:52.888653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:52.888781 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:52.889361 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:52.889366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:52.889510 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 00:45:52.889679 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 00:45:52.890028 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":152,"spinlock_wait_cycles":25856,"thread_start_us":147,"threads_started":1}
I20260927 00:45:52.893445 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:52.894169 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.893941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55380)
0927 00:45:52.894049 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:52.894112 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:45:52.894356 22686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
I20260927 00:45:52.895119 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.894920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55390)
0927 00:45:52.895022 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:52.895073 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:45:52.895282 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 00:45:52.903719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39111
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:52.944531 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } attempt: 1
I20260927 00:45:52.945467 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.945123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55402)
0927 00:45:52.945300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:52.945374 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:45:52.946224 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:52.945996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55410)
0927 00:45:52.946113 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:52.946163 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 00:45:52.946377 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
W20260927 00:45:53.009969 22743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:53.010286 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:53.010388 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:53.010483 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:53.014485 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:53.014554 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:53.014591 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:53.014626 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:53.014660 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:53.014557 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } attempt: 2
I20260927 00:45:53.015451 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.015106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55414)
0927 00:45:53.015272 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.015344 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:45:53.016181 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.015958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55418)
0927 00:45:53.016065 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.016115 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:45:53.016331 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39111: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39111: connect: Connection refused (error 111)
I20260927 00:45:53.020224 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:53.021224 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:53.022135 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:53.030167 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:53.030300 22743 server_base.cc:1034] running on GCE node
W20260927 00:45:53.030153 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:53.030174 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:53.030947 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:53.031971 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:53.033149 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790469953033136 us; error 25 us; skew 500 ppm
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:53.035687 22743 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:45:53.036547 22743 webserver.cc:460] Webserver started at http://127.0.98.61:36439/ using document root <none> and password file <none>
I20260927 00:45:53.037137 22743 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:53.037225 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:53.037442 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:53.039336 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data/instance:
uuid: "416e9b9bd5d640849dd8b828fdd3a529"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "5c39f02f159cb0afc990f6c347d98c66"
server_key_iv: "6c71a2fa96ad35b70645906128ee1b63"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.039884 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal/instance:
uuid: "416e9b9bd5d640849dd8b828fdd3a529"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "5c39f02f159cb0afc990f6c347d98c66"
server_key_iv: "6c71a2fa96ad35b70645906128ee1b63"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.043282 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 00:45:53.045552 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:53.046657 22743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 00:45:53.046798 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "416e9b9bd5d640849dd8b828fdd3a529"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "5c39f02f159cb0afc990f6c347d98c66"
server_key_iv: "6c71a2fa96ad35b70645906128ee1b63"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.046911 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:53.059602 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:53.062275 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:53.062474 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:53.069175 22743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39111
I20260927 00:45:53.069450 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39111 every 8 connection(s)
I20260927 00:45:53.070086 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:45:53.072647 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:53.076381 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:53.079245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22743
I20260927 00:45:53.079394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-1/wal/instance
I20260927 00:45:53.079680 392 external_mini_cluster.cc:1428] Setting key 7613da053fb69a85e3badce96df3a64c
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 00:45:53.090090 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.078161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.0.98.62:36299)
0927 00:45:53.078347 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.078353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.079070 (+ 717us) server_negotiation.cc:407] Connection header received
0927 00:45:53.080080 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.080084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.080139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.080245 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.081579 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.082093 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.083201 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.083444 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.083570 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.085873 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.085876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.085878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.085903 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.087408 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.087905 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.087910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.087912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.087997 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.088333 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.088338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.088340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.088528 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.088643 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.089820 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.089946 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 00:45:53.090469 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.078502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.0.1:46692)
0927 00:45:53.078902 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.078942 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.079875 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.080404 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.080410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.080723 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.081405 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.081424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.082207 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.082210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.083070 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.083079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.083600 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.083604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.083766 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.084321 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.084337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.085703 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.087536 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.087541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.087554 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.087800 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.088097 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.088100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.088102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.088237 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.088641 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.088648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.088943 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.089154 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.089497 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260927 00:45:53.091931 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:45:53.102157 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.092447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55420)
0927 00:45:53.092585 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.092596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.092676 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.093181 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.093184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.093345 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.093420 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.093425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.094358 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.094360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.094864 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.094871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.095445 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.095448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.095618 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.096049 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.096063 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.099048 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.100629 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.100634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.100638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.100820 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.101131 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.101135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.101137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.101221 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.101655 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.101657 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.101737 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.101854 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.101971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":57}
I20260927 00:45:53.102213 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.092545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.0.98.61:39111)
0927 00:45:53.092815 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.092824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.092840 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:45:53.092870 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.092873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.093067 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.093203 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.093528 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.094230 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.094975 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.095333 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.095447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.099184 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.099192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.099201 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.099220 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.100520 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.100949 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.100956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.100958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.101036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.101315 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.101318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.101320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.101543 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.101678 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.101933 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.102053 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:45:53.104177 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:53.104615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41289
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:53.104997 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.104660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37734)
0927 00:45:53.104803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.104881 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 00:45:53.105198 22760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
I20260927 00:45:53.106148 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.105951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37742)
0927 00:45:53.106057 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.106100 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:45:53.106359 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
I20260927 00:45:53.107493 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } attempt: 3
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:45:53.116541 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.107922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55436)
0927 00:45:53.108024 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.108038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.108117 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.108417 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.108419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.108557 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.108648 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.108653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.109862 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.109864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.110384 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.110391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.111106 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.111108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.111185 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.111641 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.111652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.113112 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.115128 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.115131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.115132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.115318 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.115625 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.115627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.115628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.115734 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.116129 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.116132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.116218 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.116317 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.116427 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 00:45:53.116640 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.107979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.0.98.61:39111)
0927 00:45:53.108117 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.108122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.108134 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:53.108235 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.108239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.108299 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.108406 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.108769 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.109711 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.110500 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.110978 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.111099 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.113249 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.113255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.113258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.113288 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.115018 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.115417 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.115424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.115428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.115522 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.115838 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.115842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.115843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.116036 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.116134 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.116388 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.116494 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 00:45:53.117619 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:53.118263 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.118040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37750)
0927 00:45:53.118149 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.118198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 00:45:53.118953 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.118685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37754)
0927 00:45:53.118839 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.118892 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 00:45:53.119103 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
I20260927 00:45:53.150233 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 1
I20260927 00:45:53.151108 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.150783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37756)
0927 00:45:53.150936 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.151021 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 00:45:53.151885 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.151660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37768)
0927 00:45:53.151772 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.151822 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:45:53.152026 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
I20260927 00:45:53.155558 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 1
I20260927 00:45:53.156193 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.155962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37778)
0927 00:45:53.156082 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.156132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:45:53.156996 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.156713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37790)
0927 00:45:53.156828 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.156877 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 00:45:53.157119 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
W20260927 00:45:53.214222 22818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:53.214524 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:53.214596 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:53.214674 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:53.218650 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:53.218727 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:53.218777 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:53.218834 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:53.218870 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:53.224036 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36299,127.0.98.61:39111,127.0.98.60:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:53.225049 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:53.225266 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 2
I20260927 00:45:53.226017 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:53.226051 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.225769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37798)
0927 00:45:53.225908 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.225974 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 00:45:53.226833 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.226564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37806)
0927 00:45:53.226680 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.226729 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 00:45:53.227044 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
I20260927 00:45:53.227178 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 2
I20260927 00:45:53.227837 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.227549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37814)
0927 00:45:53.227700 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.227759 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 00:45:53.228432 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.228220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37824)
0927 00:45:53.228323 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.228371 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:45:53.228567 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41289: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41289: connect: Connection refused (error 111)
W20260927 00:45:53.234575 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:53.234571 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:53.234748 22818 server_base.cc:1034] running on GCE node
W20260927 00:45:53.234763 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:53.235443 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:53.236399 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:53.237579 22818 hybrid_clock.cc:648] HybridClock initialized: now 1790469953237565 us; error 25 us; skew 500 ppm
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:53.240052 22818 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:45:53.240953 22818 webserver.cc:460] Webserver started at http://127.0.98.60:35191/ using document root <none> and password file <none>
I20260927 00:45:53.241458 22818 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:53.241540 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:53.241752 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:53.243592 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data/instance:
uuid: "4025702a9bbf43fdadec028620142439"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "c06ca93ee648c6fee80c7ab388617709"
server_key_iv: "8f74b16031d69d470106fd03eafd96aa"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.244117 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal/instance:
uuid: "4025702a9bbf43fdadec028620142439"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "c06ca93ee648c6fee80c7ab388617709"
server_key_iv: "8f74b16031d69d470106fd03eafd96aa"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.247478 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 00:45:53.249670 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:53.250958 22818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 00:45:53.251049 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "4025702a9bbf43fdadec028620142439"
format_stamp: "Formatted at 2026-09-27 00:45:53 on dist-test-slave-pkws"
server_key: "c06ca93ee648c6fee80c7ab388617709"
server_key_iv: "8f74b16031d69d470106fd03eafd96aa"
server_key_version: "encryptionkey@0"
I20260927 00:45:53.251121 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:53.270563 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:53.273334 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:53.273475 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:53.280031 22818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41289
I20260927 00:45:53.280308 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41289 every 8 connection(s)
I20260927 00:45:53.281028 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:45:53.283583 22886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:53.287123 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:53.291287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 22818
I20260927 00:45:53.291437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790469678562534-392-0/minicluster-data/master-2/wal/instance
I20260927 00:45:53.291704 392 external_mini_cluster.cc:1428] Setting key ea468314cc62ecd4c2265099a24b5d23
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:53.300725 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.288882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.0.98.62:36299)
0927 00:45:53.289030 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.289034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.289786 (+ 752us) server_negotiation.cc:407] Connection header received
0927 00:45:53.290700 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.290703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.290755 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.290866 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.292079 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.292748 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.293892 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.294120 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.294235 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.296729 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.296732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.296734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.296757 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.298336 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.299033 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.299037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.299041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.299122 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.299546 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.299549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.299550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.299730 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.299811 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.300408 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.300532 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:45:53.301570 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.289222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36299 (local address 127.0.0.1:46700)
0927 00:45:53.289642 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.289682 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.290498 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.291034 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.291040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.291358 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.291940 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.291954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.292877 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.292880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.293762 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.293771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.294266 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.294270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.294441 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.295046 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.295063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.296576 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.298453 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.298458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.298473 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.298909 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.299217 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.299220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.299223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.299408 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.299835 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.299841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.300138 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.300341 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.300603 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260927 00:45:53.302426 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.292324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.0.98.60:41289)
0927 00:45:53.292685 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.292696 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.292721 (+ 25us) server_negotiation.cc:407] Connection header received
0927 00:45:53.292778 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.292783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.292999 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.293163 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.293874 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.294598 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.295433 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.295774 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.295886 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.298528 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.298538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.298551 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.298579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.300246 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.300708 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.300713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.300715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.300796 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.301168 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.301171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.301172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.301407 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.301562 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.302113 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.302266 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260927 00:45:53.303066 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:53.311383 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.303551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39111 (local address 127.0.0.1:55440)
0927 00:45:53.303662 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.303677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.303755 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.304117 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.304119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.304279 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.304401 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.304408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.305145 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.305149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.305913 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.305923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.306368 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.306370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.306446 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.306874 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.306884 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.308088 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.309958 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.309961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.309964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.310207 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.310481 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.310484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.310486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.310582 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.310935 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.310939 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.311037 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.311123 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.311210 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 00:45:53.311758 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.303738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.0.98.61:39111)
0927 00:45:53.303892 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.303903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.303915 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:53.303950 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.303951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.303986 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.304078 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.304512 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.305050 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.306046 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.306251 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.306353 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.308217 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.308220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.308222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.308246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.309844 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.310321 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.310324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.310325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.310391 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.310671 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.310673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.310675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.310835 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.310910 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.311206 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.311335 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 00:45:53.312405 22886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:53.314571 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.304483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.0.98.62:36299)
0927 00:45:53.305195 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.305198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.305207 (+ 9us) server_negotiation.cc:407] Connection header received
0927 00:45:53.305233 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.305235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.305293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.305350 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.306445 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.307051 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.307906 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.308254 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.308328 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.311031 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.311033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.311035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.311055 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.312813 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.313437 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.313441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.313446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.313533 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.313798 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.313801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.313803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.313998 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.314210 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.314289 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.314394 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648}
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:53.318220 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 3
I20260927 00:45:53.321188 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.312914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37834)
0927 00:45:53.313050 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.313061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.313112 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.313338 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.313339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.313507 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.313608 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.313617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.314531 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.314534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.315085 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.315093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.315622 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.315625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.315712 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.316160 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.316173 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.317363 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.319519 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.319521 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.319523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.319773 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.320051 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.320055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.320059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.320175 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.320605 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.320610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.320724 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.320858 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.321014 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 00:45:53.321328 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.312989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.0.98.60:41289)
0927 00:45:53.313102 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.313106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.313119 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:53.313161 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.313167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.313212 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.313300 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.313720 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.314424 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.315199 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.315508 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.315625 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.317491 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.317495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.317498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.317521 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.319420 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.319873 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.319876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.319879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.319965 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.320307 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.320310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.320312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.320492 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.320585 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.321001 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.321116 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":42}
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:53.325227 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: Bootstrap starting.
I20260927 00:45:53.325814 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.315721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.0.98.61:39111)
0927 00:45:53.315847 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.315851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.315864 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:53.315936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.315939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.315976 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.316068 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.316649 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.317241 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.318294 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.318621 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.318713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.322500 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.322503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.322506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.322530 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.324100 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.324501 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.324504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.324506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.324584 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.324857 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.324861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.324863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.325108 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.325369 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.325532 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.325645 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 00:45:53.327716 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } attempt: 3
I20260927 00:45:53.327845 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.318577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37836)
0927 00:45:53.318689 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.318700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.318758 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.319253 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.319257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.319427 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.319533 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.319541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.321019 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.321022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.321732 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.321741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.322354 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.322357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.322446 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.322903 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.322923 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.324340 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.326320 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.326323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.326325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.326562 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.326867 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.326870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.326872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.326962 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.327372 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.327376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.327467 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.327581 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.327713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:45:53.328190 22886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:53.328496 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.318741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.0.98.60:41289)
0927 00:45:53.319053 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.319059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.319068 (+ 9us) server_negotiation.cc:407] Connection header received
0927 00:45:53.319093 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.319095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.319124 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.319192 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.319639 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.320867 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.321854 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.322207 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.322635 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.324479 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.324482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.324484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.324506 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.326213 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.326667 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.326672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.326674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.326779 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.327062 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.327065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.327066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.327259 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.327366 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.327643 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.328319 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260927 00:45:53.329120 22886 log.cc:824] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:53.329569 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:53.331331 22886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: No bootstrap required, opened a new log
I20260927 00:45:53.331387 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: Bootstrap starting.
I20260927 00:45:53.330863 22710 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "33e8f6c719fc4bf487d82abd9de9587a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46704
Sep 27 00:45:53 dist-test-slave-pkws krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:53.333997 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:53.334857 22811 log.cc:824] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:53.335596 22886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.335832 22886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4025702a9bbf43fdadec028620142439, State: Initialized, Role: FOLLOWER
I20260927 00:45:53.336169 22886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.336678 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.336805 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.337265 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: No bootstrap required, opened a new log
I20260927 00:45:53.337554 22886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:53.337975 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.328248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41289 (local address 127.0.0.1:37842)
0927 00:45:53.328395 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:53.328412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:53.328488 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:53.329180 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:53.329183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:53.329357 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:53.329478 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.329486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.330551 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.330554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.331371 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:53.331381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.331942 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.331945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:53.332049 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.332511 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:53.332529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:53.334034 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:53.336094 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.336097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.336099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.336366 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.336803 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:53.336806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:53.336808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:53.336931 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:53.337440 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:53.337444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:53.337539 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 00:45:53.337658 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 00:45:53.337798 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 00:45:53.337976 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.328669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.98.60:41289)
0927 00:45:53.328960 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.328964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.328984 (+ 20us) server_negotiation.cc:407] Connection header received
0927 00:45:53.329014 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.329016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.329056 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.329619 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.329698 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.330415 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.331523 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.331810 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.332253 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.334195 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.334198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.334201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.334225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.335983 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.336468 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.336472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.336474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.336653 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.337088 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.337091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.337093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.337298 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.337433 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.337729 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.337842 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260927 00:45:53.341023 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: Bootstrap starting.
I20260927 00:45:53.341282 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.341482 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416e9b9bd5d640849dd8b828fdd3a529, State: Initialized, Role: FOLLOWER
I20260927 00:45:53.341790 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.343050 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:53.343573 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:53.344449 22736 log.cc:824] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: Log is configured to *not* fsync() on all Append() calls
W20260927 00:45:53.343310 22905 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:53.345150 22905 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:53.347031 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: No bootstrap required, opened a new log
W20260927 00:45:53.349124 22919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:53.349207 22919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:53.346421 22818 master_runner.cc:186] Error getting master registration for 127.0.98.62:36299: OK, instance_id { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" instance_seqno: 1790469952867711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:45:53.349541 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.349678 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.349702 22818 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:45:53.350023 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.350276 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e8f6c719fc4bf487d82abd9de9587a, State: Initialized, Role: FOLLOWER
I20260927 00:45:53.350646 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.351126 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.351356 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.351987 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:53.350277 22743 master_runner.cc:186] Error getting master registration for 127.0.98.62:36299: OK, instance_id { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" instance_seqno: 1790469952867711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:45:53.354498 22743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:45:53.354691 22710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:45:53.354803 22710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:53.355859 22710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.356792 22710 leader_election.cc:288] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [CANDIDATE]: Term 1 election: Requested vote from peers 416e9b9bd5d640849dd8b828fdd3a529 (127.0.98.61:39111), 4025702a9bbf43fdadec028620142439 (127.0.98.60:41289)
I20260927 00:45:53.357517 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e8f6c719fc4bf487d82abd9de9587a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4025702a9bbf43fdadec028620142439"
I20260927 00:45:53.357759 22860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:53.357759 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33e8f6c719fc4bf487d82abd9de9587a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "416e9b9bd5d640849dd8b828fdd3a529"
I20260927 00:45:53.357972 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:53.359084 22860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e8f6c719fc4bf487d82abd9de9587a in term 1.
I20260927 00:45:53.359215 22785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e8f6c719fc4bf487d82abd9de9587a in term 1.
I20260927 00:45:53.359934 22685 leader_election.cc:302] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33e8f6c719fc4bf487d82abd9de9587a, 4025702a9bbf43fdadec028620142439, 416e9b9bd5d640849dd8b828fdd3a529; no voters:
I20260927 00:45:53.360212 22921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:53.360406 22921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [term 1 LEADER]: Becoming Leader. State: Replica: 33e8f6c719fc4bf487d82abd9de9587a, State: Running, Role: LEADER
I20260927 00:45:53.360729 22921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } }
I20260927 00:45:53.362476 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e8f6c719fc4bf487d82abd9de9587a. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.362589 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: This master's current role is: LEADER
W20260927 00:45:53.364389 22689 tablet.cc:2369] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:53.367273 22668 master_runner.cc:263] No leader master found from master 33e8f6c719fc4bf487d82abd9de9587a
I20260927 00:45:53.367347 22668 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 00:45:53.370093 22932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:53.370178 22932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:53.370436 22939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:53.371280 22939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:53.380736 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.363935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.0.98.61:39111)
0927 00:45:53.364049 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.364055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.366894 (+ 2839us) server_negotiation.cc:407] Connection header received
0927 00:45:53.367063 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.367069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.367120 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.367170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.370193 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.370878 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.374156 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.374461 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.374550 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.376944 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.376947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.376950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.376994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.378809 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.379345 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.379350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.379352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.379444 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.379766 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.379769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.379772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.380008 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.380132 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.380468 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.380578 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:53.383682 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.363332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.0.98.62:36299)
0927 00:45:53.363440 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.363445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.365230 (+ 1785us) server_negotiation.cc:407] Connection header received
0927 00:45:53.365421 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.365424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.365472 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.365676 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.368092 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.368840 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.373065 (+ 4225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.373381 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.373455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.378239 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.378244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.378249 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.378279 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.379961 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.382114 (+ 2153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.382117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.382119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.382209 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.382612 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.382618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.382619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.382861 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.383003 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.383361 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.383496 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 00:45:53.386989 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:53.364593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.98.60:41289)
0927 00:45:53.364694 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.364698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.370113 (+ 5415us) server_negotiation.cc:407] Connection header received
0927 00:45:53.372028 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.372031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.372078 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.372125 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.374693 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.375173 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.380432 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:53.380761 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:53.380842 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:53.383402 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:53.383406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:53.383408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:53.383433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:53.385179 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.385654 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.385659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.385661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.385744 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:53.386060 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:53.386064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:53.386066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:53.386267 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:53.386383 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:53.386717 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 00:45:53.386836 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 00:45:53.394917 22860 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [term 1 FOLLOWER]: Refusing update from remote peer 33e8f6c719fc4bf487d82abd9de9587a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:53.395149 22785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [term 1 FOLLOWER]: Refusing update from remote peer 33e8f6c719fc4bf487d82abd9de9587a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:53.395632 22921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:53.395825 22933 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [LEADER]: Connected to new peer: Peer: permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:53.399758 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e8f6c719fc4bf487d82abd9de9587a. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.399884 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.400166 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e8f6c719fc4bf487d82abd9de9587a. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.400275 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.409946 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.410075 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.410861 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.411018 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:53.411278 22921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.411402 22921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:53.411512 22944 mvcc.cc:205] Tried to move back new op lower bound from 7333764929081614336 to 7333764928968728576. Current Snapshot: MvccSnapshot[applied={T|T < 7333764929081614336}]
I20260927 00:45:53.411825 22946 mvcc.cc:205] Tried to move back new op lower bound from 7333764929081614336 to 7333764928968728576. Current Snapshot: MvccSnapshot[applied={T|T < 7333764929081614336}]
I20260927 00:45:53.412292 22894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33e8f6c719fc4bf487d82abd9de9587a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e8f6c719fc4bf487d82abd9de9587a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36299 } } peers { permanent_uuid: "416e9b9bd5d640849dd8b828fdd3a529" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39111 } } peers { permanent_uuid: "4025702a9bbf43fdadec028620142439" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41289 } } }
I20260927 00:45:53.412410 22894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:53.415109 22947 mvcc.cc:205] Tried to move back new op lower bound from 7333764929081614336 to 7333764928968728576. Current Snapshot: MvccSnapshot[applied={T|T < 7333764929081614336}]
I20260927 00:45:53.418488 22939 catalog_manager.cc:1347] Generated new cluster ID: d186213972c54099a32e1c3d1c920b39
I20260927 00:45:53.418553 22939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:53.442340 22939 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:53.443675 22939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:53.457155 22939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e8f6c719fc4bf487d82abd9de9587a: Generated new TSK 0
I20260927 00:45:53.458014 22939 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:45:53.489388 22738 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.486515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.98.62:36299)
0927 00:45:53.486632 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.486637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.488326 (+ 1689us) server_negotiation.cc:407] Connection header received
0927 00:45:53.488517 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.488520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.488574 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.488626 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.489260 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55754: BlockingRecv error: recv got EOF from 127.0.0.1:55754 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 00:45:53.490972 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.487224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.0.98.61:39111)
0927 00:45:53.487337 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.487342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.489820 (+ 2478us) server_negotiation.cc:407] Connection header received
0927 00:45:53.490267 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.490270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.490318 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.490371 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.490851 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57228: BlockingRecv error: recv got EOF from 127.0.0.1:57228 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:45:53.491884 22891 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.487842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.0.98.60:41289)
0927 00:45:53.487948 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.487953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.490027 (+ 2074us) server_negotiation.cc:407] Connection header received
0927 00:45:53.491201 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.491205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.491251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.491301 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.491765 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37088: BlockingRecv error: recv got EOF from 127.0.0.1:37088 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 00:45:53.496119 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.494684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.0.98.61:39111)
0927 00:45:53.494774 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.494777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.495267 (+ 490us) server_negotiation.cc:407] Connection header received
0927 00:45:53.495461 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.495464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.495526 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.495573 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.496034 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57244: BlockingRecv error: recv got EOF from 127.0.0.1:57244 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 00:45:53.500669 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.499335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.0.98.61:39111)
0927 00:45:53.499432 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.499436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.499891 (+ 455us) server_negotiation.cc:407] Connection header received
0927 00:45:53.500053 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.500056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.500097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.500142 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.500590 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57258: BlockingRecv error: recv got EOF from 127.0.0.1:57258 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 00:45:53.505811 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.504390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.0.98.61:39111)
0927 00:45:53.504476 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.504480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.505039 (+ 559us) server_negotiation.cc:407] Connection header received
0927 00:45:53.505203 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.505206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.505246 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.505291 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.505733 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57264: BlockingRecv error: recv got EOF from 127.0.0.1:57264 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 00:45:53.512840 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.511480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.98.61:39111)
0927 00:45:53.511594 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.511598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.512062 (+ 464us) server_negotiation.cc:407] Connection header received
0927 00:45:53.512223 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.512226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.512266 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.512311 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.512738 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57270: BlockingRecv error: recv got EOF from 127.0.0.1:57270 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 00:45:53.524042 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.522597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.98.61:39111)
0927 00:45:53.522690 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.522694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.523200 (+ 506us) server_negotiation.cc:407] Connection header received
0927 00:45:53.523373 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.523376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.523420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.523467 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.523951 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57286: BlockingRecv error: recv got EOF from 127.0.0.1:57286 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 00:45:53.543998 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.542261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.0.98.61:39111)
0927 00:45:53.542386 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.542391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.542996 (+ 605us) server_negotiation.cc:407] Connection header received
0927 00:45:53.543214 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.543218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.543271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.543327 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.543888 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57290: BlockingRecv error: recv got EOF from 127.0.0.1:57290 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 00:45:53.579671 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.578433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.98.61:39111)
0927 00:45:53.578596 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.578601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.578773 (+ 172us) server_negotiation.cc:407] Connection header received
0927 00:45:53.578954 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.578958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.579011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.579105 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.579565 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 00:45:53.647248 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.645979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.0.98.61:39111)
0927 00:45:53.646159 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.646164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.646341 (+ 177us) server_negotiation.cc:407] Connection header received
0927 00:45:53.646521 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.646525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.646580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.646671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.647133 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57306: BlockingRecv error: recv got EOF from 127.0.0.1:57306 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 00:45:53.778281 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:53.777208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.0.98.61:39111)
0927 00:45:53.777385 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:53.777388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:53.777469 (+ 81us) server_negotiation.cc:407] Connection header received
0927 00:45:53.777593 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:53.777596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:53.777645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:53.777720 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:53.778156 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57308: BlockingRecv error: recv got EOF from 127.0.0.1:57308 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 00:45:54.037289 22816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:54.036181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.0.98.61:39111)
0927 00:45:54.036341 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.036345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.036384 (+ 39us) server_negotiation.cc:407] Connection header received
0927 00:45:54.036529 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.036531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.036580 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.036651 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:54.037138 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57318: BlockingRecv error: recv got EOF from 127.0.0.1:57318 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 00:45:54.346524 22905 catalog_manager.cc:1259] Loaded cluster ID: d186213972c54099a32e1c3d1c920b39
I20260927 00:45:54.346607 22905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: loading cluster ID for follower catalog manager: success
I20260927 00:45:54.348312 22905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: acquiring CA information for follower catalog manager: success
I20260927 00:45:54.349653 22905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4025702a9bbf43fdadec028620142439: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:45:54.350313 22919 catalog_manager.cc:1259] Loaded cluster ID: d186213972c54099a32e1c3d1c920b39
I20260927 00:45:54.350365 22919 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: loading cluster ID for follower catalog manager: success
I20260927 00:45:54.352437 22919 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: acquiring CA information for follower catalog manager: success
I20260927 00:45:54.353582 22919 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 416e9b9bd5d640849dd8b828fdd3a529: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:45:54.555371 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:54.551278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.0.98.61:39111)
0927 00:45:54.551584 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.551588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.551605 (+ 17us) server_negotiation.cc:407] Connection header received
0927 00:45:54.551768 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.551771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.551894 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.551998 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:54.552232 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.552869 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.553790 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.554072 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.554153 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:54.554416 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:54.554941 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:54.555081 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:54.555109 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 00:45:54.555155 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260927 00:45:54.561614 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:54.558386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.98.60:41289)
0927 00:45:54.558597 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.558600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.558613 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:54.558683 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.558685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.558785 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.558876 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:54.559048 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.559538 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.560279 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.560494 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.560565 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:54.560802 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:54.561284 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:54.561375 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:54.561448 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 00:45:54.561504 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260927 00:45:54.568612 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:54.564305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.98.61:39111)
0927 00:45:54.564421 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.564425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.565036 (+ 611us) server_negotiation.cc:407] Connection header received
0927 00:45:54.565125 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.565131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.565179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.565244 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:54.565577 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.566302 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.567458 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.567764 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.567864 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:54.568081 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:54.568158 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:54.568230 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:54.568424 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 00:45:54.568487 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:54.568739 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:54.564502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.0.98.62:36299)
0927 00:45:54.564720 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.564723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.564736 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:54.564848 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.564853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.565035 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.565153 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:54.565396 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.566222 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.567199 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.567466 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.567569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:54.567869 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:54.568195 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:54.568373 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:54.568507 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 00:45:54.568573 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260927 00:45:54.568943 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:54.564832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.98.60:41289)
0927 00:45:54.564978 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:54.564984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:54.565315 (+ 331us) server_negotiation.cc:407] Connection header received
0927 00:45:54.565427 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:54.565431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:54.565476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:54.565556 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:54.565906 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.566486 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.567611 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:54.567931 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:54.568011 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:54.568434 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:54.568529 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:54.568595 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:54.568732 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 00:45:54.568787 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 00:45:54.575848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22668
I20260927 00:45:54.582091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22743
I20260927 00:45:54.587697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 22818
2026-09-27T00:45:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3093 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 00:45:54 dist-test-slave-pkws krb5kdc[23034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 00:45:54 dist-test-slave-pkws krb5kdc[23034](info): set up 2 sockets
Sep 27 00:45:54 dist-test-slave-pkws krb5kdc[23034](info): commencing operation
krb5kdc: starting...
W20260927 00:45:55.621269 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/test-user.keytab.
Sep 27 00:45:55 dist-test-slave-pkws krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T00:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T00:45:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 00:45:55.740160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 00:45:55.849871 23047 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:55.850122 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:55.850175 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:55.850242 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:55.854667 23047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:55.854715 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:55.854735 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:55.854753 23047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:55.854803 23047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:55.859822 23047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:55.860792 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:55.861761 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 00:45:55.869854 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:55.869892 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:55.870208 23047 server_base.cc:1034] running on GCE node
W20260927 00:45:55.869879 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:55.870909 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:55.872046 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:55.873202 23047 hybrid_clock.cc:648] HybridClock initialized: now 1790469955873182 us; error 33 us; skew 500 ppm
Sep 27 00:45:55 dist-test-slave-pkws krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:55.875936 23047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 00:45:55.876788 23047 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260927 00:45:55.877379 23047 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:55.877472 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:55.877712 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:55.879680 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data/instance:
uuid: "580d135f8a044775be2c07c63d72dfee"
format_stamp: "Formatted at 2026-09-27 00:45:55 on dist-test-slave-pkws"
server_key: "24df876be8cd03595ccb9b7405816076"
server_key_iv: "14c52de2b16354d2c1c415342d66eec6"
server_key_version: "encryptionkey@0"
I20260927 00:45:55.880239 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance:
uuid: "580d135f8a044775be2c07c63d72dfee"
format_stamp: "Formatted at 2026-09-27 00:45:55 on dist-test-slave-pkws"
server_key: "24df876be8cd03595ccb9b7405816076"
server_key_iv: "14c52de2b16354d2c1c415342d66eec6"
server_key_version: "encryptionkey@0"
I20260927 00:45:55.883759 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 00:45:55.886132 23062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:55.887283 23047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:55.887421 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal
uuid: "580d135f8a044775be2c07c63d72dfee"
format_stamp: "Formatted at 2026-09-27 00:45:55 on dist-test-slave-pkws"
server_key: "24df876be8cd03595ccb9b7405816076"
server_key_iv: "14c52de2b16354d2c1c415342d66eec6"
server_key_version: "encryptionkey@0"
I20260927 00:45:55.887535 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:55.903697 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:55.906931 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:55.907135 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:55.913753 23047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40491
I20260927 00:45:55.914029 23114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40491 every 8 connection(s)
I20260927 00:45:55.914697 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/data/info.pb
I20260927 00:45:55.915787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 23047
I20260927 00:45:55.915876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-0/wal/instance
I20260927 00:45:55.916183 392 external_mini_cluster.cc:1428] Setting key 0ef5ad41c2e7297376e1b15e2fab4a5c
I20260927 00:45:55.917537 23115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:55.922217 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 00:45:55 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:55.936730 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.924287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:60748)
0927 00:45:55.924723 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:55.924763 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:55.925616 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:55.926344 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:55.926348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:55.926714 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:55.927067 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:55.927090 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:55.928217 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:55.928220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:55.929015 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:55.929023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:55.929636 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:55.929639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:55.929810 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:55.930470 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:55.930489 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:55.931865 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:55.933761 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:55.933770 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:55.933779 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:55.933992 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:55.934295 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:55.934298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:55.934299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:55.934421 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:55.935046 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:55.935052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:55.935186 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 00:45:55.935430 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 00:45:55.935821 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":203,"thread_start_us":94,"threads_started":1}
I20260927 00:45:55.936730 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.925076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.0.98.62:40491)
0927 00:45:55.925331 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:55.925340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:55.925372 (+ 32us) server_negotiation.cc:407] Connection header received
0927 00:45:55.925877 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:55.925881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:55.926171 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:55.926319 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:55.927227 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:55.928087 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:55.929124 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:55.929482 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:55.929664 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:55.931996 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:55.932002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:55.932014 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:55.932044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:55.933660 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:55.934093 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:55.934098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:55.934100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:55.934197 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:55.934515 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:55.934518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:55.934519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:55.934936 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:55.935073 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:55.935503 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 00:45:55.935734 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":157,"spinlock_wait_cycles":2816,"thread_start_us":91,"threads_started":1}
I20260927 00:45:55.939399 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:55.940056 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.939813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44540)
0927 00:45:55.939928 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:55.939998 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 00:45:55.940272 23065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
I20260927 00:45:55.941076 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.940844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44552)
0927 00:45:55.940975 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:55.941027 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab.
W20260927 00:45:55.941282 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41823: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
I20260927 00:45:55.942039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41823
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:55.990522 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } attempt: 1
I20260927 00:45:55.991423 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.991127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44554)
0927 00:45:55.991281 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:55.991346 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 00:45:55.992169 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:55.991975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44570)
0927 00:45:55.992075 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:55.992119 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 00:45:55.992309 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41823: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
W20260927 00:45:56.047897 23122 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:56.048133 23122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:56.048156 23122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:56.048192 23122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:56.052104 23122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:56.052153 23122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:56.052172 23122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:56.052188 23122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:56.052204 23122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:56.057325 23122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:56.058255 23122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:56.059195 23122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:56.060499 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } attempt: 2
I20260927 00:45:56.061323 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.061049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44578)
0927 00:45:56.061190 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.061253 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:45:56.062093 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.061909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44592)
0927 00:45:56.062003 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.062045 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 00:45:56.062238 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41823: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41823: connect: Connection refused (error 111)
W20260927 00:45:56.067508 23130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:56.067510 23128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:56.067528 23127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:56.067747 23122 server_base.cc:1034] running on GCE node
I20260927 00:45:56.068230 23122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:56.069288 23122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:56.070446 23122 hybrid_clock.cc:648] HybridClock initialized: now 1790469956070416 us; error 42 us; skew 500 ppm
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:56.073096 23122 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 00:45:56.073822 23122 webserver.cc:460] Webserver started at http://127.0.98.61:41573/ using document root <none> and password file <none>
I20260927 00:45:56.074293 23122 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:56.074361 23122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:56.074510 23122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:56.076319 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data/instance:
uuid: "7f3874208d2345e9bafa9df2e110360f"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "c604a88c5af732ac3b3a3f38b6762857"
server_key_iv: "8705a83d93f68bd15078fa3d8c60b7d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.076870 23122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal/instance:
uuid: "7f3874208d2345e9bafa9df2e110360f"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "c604a88c5af732ac3b3a3f38b6762857"
server_key_iv: "8705a83d93f68bd15078fa3d8c60b7d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.080178 23122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 00:45:56.082377 23137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:56.083503 23122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:56.083640 23122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal
uuid: "7f3874208d2345e9bafa9df2e110360f"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "c604a88c5af732ac3b3a3f38b6762857"
server_key_iv: "8705a83d93f68bd15078fa3d8c60b7d7"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.083742 23122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:56.093215 23122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:56.095623 23122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:56.095803 23122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:56.102267 23122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41823
I20260927 00:45:56.102522 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41823 every 8 connection(s)
I20260927 00:45:56.103145 23122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/data/info.pb
I20260927 00:45:56.105701 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:56.107286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 23122
I20260927 00:45:56.107391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-1/wal/instance
I20260927 00:45:56.107682 392 external_mini_cluster.cc:1428] Setting key ec2e82a670dd1886111015129c5c027d
I20260927 00:45:56.109458 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:56.124053 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.111165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.98.62:40491)
0927 00:45:56.111343 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.111350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.112174 (+ 824us) server_negotiation.cc:407] Connection header received
0927 00:45:56.113542 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.113545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.113592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.113657 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.115149 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.115717 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.116923 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.117175 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.117256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.119940 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.119950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.119953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.119976 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.121406 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.122011 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.122014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.122015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.122093 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.122469 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.122471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.122472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.122634 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.122721 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.123824 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.123926 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 00:45:56.124727 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.111506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:60762)
0927 00:45:56.111985 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.112044 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.113296 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.113868 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.113877 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.114346 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.114993 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.115015 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.115849 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.115854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.116777 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.116786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.117295 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.117298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.117430 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.117950 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.117965 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.119769 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.121523 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.121531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.121547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.121906 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.122189 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.122195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.122200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.122377 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.122756 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.122765 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.123096 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.123311 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.123638 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260927 00:45:56.126083 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 00:45:56.134352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskql8uh0/build/debug/bin/kudu
/tmp/dist-test-taskql8uh0/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35217
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 00:45:56.135548 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.126562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44606)
0927 00:45:56.126707 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.126718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.126773 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.127279 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.127281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.127457 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.127535 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.127540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.128534 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.128540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.129109 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.129117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.129687 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.129690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.129834 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.130286 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.130297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.131555 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.133735 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.133738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.133740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.133996 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.134287 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.134291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.134293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.134391 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.134889 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.134895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.135017 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.135137 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.135305 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":41}
I20260927 00:45:56.135627 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.126662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.0.98.61:41823)
0927 00:45:56.126972 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.126979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.126998 (+ 19us) server_negotiation.cc:407] Connection header received
0927 00:45:56.127027 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.127030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.127162 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.127299 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.127644 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.128407 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.129217 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.129569 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.129681 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.131698 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.131706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.131714 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.131734 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.133622 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.134112 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.134116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.134117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.134187 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.134491 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.134496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.134499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.134780 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.134891 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.135204 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.135393 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":238,"thread_start_us":140,"threads_started":1}
I20260927 00:45:56.137404 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:56.138134 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.137886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49392)
0927 00:45:56.138014 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.138083 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 00:45:56.138350 23141 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
I20260927 00:45:56.139122 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.138906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49394)
0927 00:45:56.139023 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.139066 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 00:45:56.139281 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
I20260927 00:45:56.153403 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } attempt: 3
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 00:45:56.161692 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.153946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.0.98.61:41823)
0927 00:45:56.154057 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.154061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.154077 (+ 16us) server_negotiation.cc:407] Connection header received
0927 00:45:56.154203 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.154206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.154250 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.154345 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.154723 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.155480 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.156217 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.156566 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.156668 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.158796 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.158799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.158801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.158821 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.160217 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.160660 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.160663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.160664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.160732 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.161018 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.161020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.161021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.161174 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.161248 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.161511 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.161595 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:45:56.161731 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.153870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44612)
0927 00:45:56.154013 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.154037 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.154115 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.154363 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.154365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.154521 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.154620 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.154625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.155601 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.155603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.156116 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.156123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.156673 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.156675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.156752 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.157269 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.157281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.158669 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.160327 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.160330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.160332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.160567 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.160816 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.160819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.160821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.160938 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.161261 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.161264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.161356 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.161462 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.161583 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 00:45:56.162751 23115 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:56.163399 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.163184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49406)
0927 00:45:56.163285 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.163348 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 00:45:56.164088 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.163857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49410)
0927 00:45:56.163995 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.164041 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 00:45:56.164229 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
I20260927 00:45:56.188470 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 1
I20260927 00:45:56.189213 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.188956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49426)
0927 00:45:56.189096 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.189150 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 00:45:56.189879 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.189676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49434)
0927 00:45:56.189779 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.189833 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:45:56.190006 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
I20260927 00:45:56.195372 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 1
I20260927 00:45:56.195971 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.195747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49448)
0927 00:45:56.195862 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.195909 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 00:45:56.196668 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.196434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49452)
0927 00:45:56.196535 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.196586 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 00:45:56.196811 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
W20260927 00:45:56.243268 23196 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 00:45:56.243496 23196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 00:45:56.243520 23196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 00:45:56.243556 23196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 00:45:56.247396 23196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 00:45:56.247444 23196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 00:45:56.247463 23196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 00:45:56.247480 23196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 00:45:56.247495 23196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 00:45:56.252333 23196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:41823,127.0.98.60:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 00:07:42 UTC on ccycloud.quasar-euhhvn.root.comops.site
build id 4675561
I20260927 00:45:56.253157 23196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 00:45:56.253996 23196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 00:45:56.258131 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 2
I20260927 00:45:56.258894 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.258576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49468)
0927 00:45:56.258737 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.258805 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 00:45:56.259532 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.259320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49484)
0927 00:45:56.259425 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.259474 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 00:45:56.259658 23190 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
W20260927 00:45:56.262319 23202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:56.262429 23196 server_base.cc:1034] running on GCE node
W20260927 00:45:56.262279 23205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 00:45:56.262297 23203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 00:45:56.262984 23196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 00:45:56.263988 23196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 00:45:56.265168 23196 hybrid_clock.cc:648] HybridClock initialized: now 1790469956265154 us; error 22 us; skew 500 ppm
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 00:45:56.267853 23196 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 00:45:56.268637 23196 webserver.cc:460] Webserver started at http://127.0.98.60:36037/ using document root <none> and password file <none>
I20260927 00:45:56.269213 23196 fs_manager.cc:359] Metadata directory not provided
I20260927 00:45:56.269302 23196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 00:45:56.269526 23196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 00:45:56.271400 23196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data/instance:
uuid: "c59afec6f0f048dd899c64916c12e9d8"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "bfa47a5bc2bd4cbe8139ff6d57b2cbf4"
server_key_iv: "ec7020738de2833b52fd0f186c1aecef"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.271948 23196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal/instance:
uuid: "c59afec6f0f048dd899c64916c12e9d8"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "bfa47a5bc2bd4cbe8139ff6d57b2cbf4"
server_key_iv: "ec7020738de2833b52fd0f186c1aecef"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.271999 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 2
I20260927 00:45:56.272692 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.272438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49492)
0927 00:45:56.272561 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.272609 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 00:45:56.273427 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.273222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49506)
0927 00:45:56.273325 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.273375 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 00:45:56.273603 23115 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35217: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35217: connect: Connection refused (error 111)
I20260927 00:45:56.275913 23196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 00:45:56.278177 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 00:45:56.279253 23196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 00:45:56.279386 23196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal
uuid: "c59afec6f0f048dd899c64916c12e9d8"
format_stamp: "Formatted at 2026-09-27 00:45:56 on dist-test-slave-pkws"
server_key: "bfa47a5bc2bd4cbe8139ff6d57b2cbf4"
server_key_iv: "ec7020738de2833b52fd0f186c1aecef"
server_key_version: "encryptionkey@0"
I20260927 00:45:56.279512 23196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 00:45:56.318581 23196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 00:45:56.321897 23196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 00:45:56.322101 23196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 00:45:56.329115 23196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35217
I20260927 00:45:56.329351 23264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35217 every 8 connection(s)
I20260927 00:45:56.330090 23196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/data/info.pb
I20260927 00:45:56.330798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskql8uh0/build/debug/bin/kudu as pid 23196
I20260927 00:45:56.330931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskql8uh0/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790469678562534-392-0/minicluster-data/master-2/wal/instance
I20260927 00:45:56.331235 392 external_mini_cluster.cc:1428] Setting key 958e5071e8976694ab13d5477d98e1de
I20260927 00:45:56.333309 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 00:45:56.336814 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 00:45:56.345909 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.332573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.0.98.60:35217)
0927 00:45:56.333064 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.333084 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.333219 (+ 135us) server_negotiation.cc:407] Connection header received
0927 00:45:56.334018 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.334035 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.334372 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.334710 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.335476 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.336465 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.337418 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.337920 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.338102 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.340815 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.340820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.340829 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.340854 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.342653 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.343134 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.343139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.343145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.343310 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.343604 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.343609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.343611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.344088 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.344307 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.344550 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.344840 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260927 00:45:56.347610 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.338492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.98.62:40491)
0927 00:45:56.338625 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.338628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.339028 (+ 400us) server_negotiation.cc:407] Connection header received
0927 00:45:56.339140 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.339145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.339213 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.339314 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.340012 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.340542 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.341496 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.341857 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.342002 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.344380 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.344384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.344387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.344411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.345963 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.346461 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.346464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.346465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.346538 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.346809 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.346812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.346813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.347040 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.347117 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.347418 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.347506 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:56.347733 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.338442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:60768)
0927 00:45:56.338884 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.338931 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.339044 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.339355 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.339359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.339664 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.339870 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.339878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.340663 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.340666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.341372 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.341381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.342007 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.342010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.342091 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.342541 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.342553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.344241 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.346089 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.346097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.346102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.346363 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.346625 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.346628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.346630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.346719 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.347117 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.347121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.347232 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.347369 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.347525 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":111,"threads_started":1}
I20260927 00:45:56.348742 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:56.350804 23190 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 3
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:56.358258 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.348377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.98.62:40491)
0927 00:45:56.348784 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.348787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.348797 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:45:56.348847 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.348850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.348893 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.349296 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.349685 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.350357 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.351254 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.351569 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.351663 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.354375 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.354381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.354386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.354415 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.356132 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.356541 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.356544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.356546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.356645 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.356975 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.356979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.356988 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.357229 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.357444 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.357870 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.358101 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260927 00:45:56.358501 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.349287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.98.61:41823)
0927 00:45:56.349402 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.349407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.349524 (+ 117us) server_negotiation.cc:407] Connection header received
0927 00:45:56.349657 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.349659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.349709 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.349794 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.350119 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.350720 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.351736 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.352089 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.352192 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.354811 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.354816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.354820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.354850 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.356570 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.357067 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.357071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.357073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.357157 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.357485 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.357490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.357493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.357716 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.357804 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.358188 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.358307 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:56.358803 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.349172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41823 (local address 127.0.0.1:44616)
0927 00:45:56.349469 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.349481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.349557 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.349813 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.349815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.349934 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.350013 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.350017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.350822 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.350825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.351591 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.351603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.352194 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.352198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.352283 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.352999 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.353013 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.354688 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.356668 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.356674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.356679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.356945 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.357249 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.357253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.357255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.357358 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.357881 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.357884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.357967 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.358106 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.358220 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260927 00:45:56.359673 23265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
I20260927 00:45:56.359860 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.351189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49528)
0927 00:45:56.351336 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.351350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.351426 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.351641 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.351643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.351816 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.351888 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.351893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.352871 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.352874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.353670 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.353679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.354173 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.354177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.354254 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.354908 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.354922 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.356355 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.358304 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.358307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.358309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.358557 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.358878 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.358883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.358884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.358971 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.359368 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.359371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.359460 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.359573 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.359698 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 00:45:56.360395 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.351234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.98.60:35217)
0927 00:45:56.351326 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.351330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.351390 (+ 60us) server_negotiation.cc:407] Connection header received
0927 00:45:56.351492 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.351496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.351534 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.351629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.352006 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.352765 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.353786 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.354063 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.354171 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.356519 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.356523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.356527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.356554 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.358186 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.358688 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.358690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.358692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.358783 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.359081 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.359085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.359086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.359257 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.359369 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.360121 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.360223 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 00:45:56.363896 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: Bootstrap starting.
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:56.366770 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:56.367693 23190 log.cc:824] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:56.368428 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.359581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.0.98.61:41823)
0927 00:45:56.359684 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.359688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.359700 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:56.359732 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.359734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.359769 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.359850 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.360802 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.361456 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.362433 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.362729 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.362844 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.365079 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.365084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.365087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.365116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.366570 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.367062 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.367065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.367067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.367147 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.367420 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.367423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.367425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.367665 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.367860 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.368034 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.368158 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 00:45:56.369832 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: No bootstrap required, opened a new log
I20260927 00:45:56.370623 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.360091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49534)
0927 00:45:56.360250 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.360261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.360322 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.360727 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.360729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.360878 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.360998 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.361004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.361867 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.361871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.362580 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.362588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.363321 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.363325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.363402 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.363798 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.363810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.365409 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.368921 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.368926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.368931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.369152 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.369688 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.369690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.369691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.369759 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.370147 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.370150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.370231 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.370329 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.370457 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 00:45:56.370649 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.360174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.0.98.60:35217)
0927 00:45:56.360562 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.360565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.360575 (+ 10us) server_negotiation.cc:407] Connection header received
0927 00:45:56.360602 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.360604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.360638 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.360718 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.361103 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.361746 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.362687 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.362978 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.363091 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.365555 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.365558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.365561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.365893 (+ 332us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.368777 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.369251 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.369257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.369260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.369337 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.369858 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.369861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.369863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.370064 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.370147 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.370382 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.370511 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":319,"thread_start_us":107,"threads_started":1}
I20260927 00:45:56.371430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 00:45:56.372782 23115 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } attempt: 3
I20260927 00:45:56.374137 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: Bootstrap starting.
I20260927 00:45:56.373440 23089 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "580d135f8a044775be2c07c63d72dfee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60780
I20260927 00:45:56.375653 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.375860 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3874208d2345e9bafa9df2e110360f, State: Initialized, Role: FOLLOWER
I20260927 00:45:56.376174 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.376585 23265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:56.376869 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.377029 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.377384 23265 log.cc:824] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:56.377557 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 00:45:56 dist-test-slave-pkws krb5kdc[23034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790469955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 00:45:56.379818 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: No bootstrap required, opened a new log
I20260927 00:45:56.382105 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.373209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35217 (local address 127.0.0.1:49538)
0927 00:45:56.373324 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 00:45:56.373339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 00:45:56.373412 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 00:45:56.373811 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 00:45:56.373814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 00:45:56.373977 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 00:45:56.374056 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.374061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.374819 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.374822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.375634 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 00:45:56.375644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.376188 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.376192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 00:45:56.376289 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.376763 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 00:45:56.376778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 00:45:56.378239 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 00:45:56.380375 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.380378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.380380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.380626 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.381023 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 00:45:56.381026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 00:45:56.381028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 00:45:56.381138 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 00:45:56.381627 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 00:45:56.381630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 00:45:56.381729 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 00:45:56.381824 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 00:45:56.381947 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 00:45:56.382400 23284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:56.382470 23284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:56.382664 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.373526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.0.98.60:35217)
0927 00:45:56.373629 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.373633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.373644 (+ 11us) server_negotiation.cc:407] Connection header received
0927 00:45:56.373680 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.373681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.373710 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.373795 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.374166 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.374669 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.375775 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.376095 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.376184 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.378418 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.378421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.378424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.378448 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.380267 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.380753 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.380756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.380758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.380860 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.381264 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.381267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.381269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.381469 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.381566 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.382336 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.382458 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:56.384268 23265 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.384505 23265 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59afec6f0f048dd899c64916c12e9d8, State: Initialized, Role: FOLLOWER
I20260927 00:45:56.384881 23265 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.385697 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.386031 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.386744 23265 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:56.387897 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: Bootstrap starting.
I20260927 00:45:56.390686 23115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: Neither blocks nor log segments found. Creating new log.
I20260927 00:45:56.391510 23115 log.cc:824] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: Log is configured to *not* fsync() on all Append() calls
I20260927 00:45:56.381124 23122 master_runner.cc:186] Error getting master registration for 127.0.98.62:40491: OK, instance_id { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" instance_seqno: 1790469955911965 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:45:56.392035 23122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:45:56.393714 23115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: No bootstrap required, opened a new log
I20260927 00:45:56.390254 23196 master_runner.cc:186] Error getting master registration for 127.0.98.62:40491: OK, instance_id { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" instance_seqno: 1790469955911965 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 00:45:56.394253 23196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:45:56.396773 23115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.397017 23115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580d135f8a044775be2c07c63d72dfee, State: Initialized, Role: FOLLOWER
I20260927 00:45:56.397348 23115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.398638 23115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 00:45:56.398721 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 00:45:56.398808 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:56.400120 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.401137 23089 leader_election.cc:288] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [CANDIDATE]: Term 1 election: Requested vote from peers 7f3874208d2345e9bafa9df2e110360f (127.0.98.61:41823), c59afec6f0f048dd899c64916c12e9d8 (127.0.98.60:35217)
I20260927 00:45:56.401651 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "580d135f8a044775be2c07c63d72dfee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f3874208d2345e9bafa9df2e110360f"
I20260927 00:45:56.401739 23239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "580d135f8a044775be2c07c63d72dfee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c59afec6f0f048dd899c64916c12e9d8"
W20260927 00:45:56.401783 23297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 00:45:56.401894 23164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [term 0 FOLLOWER]: Advancing to term 1
W20260927 00:45:56.402014 23297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:56.401978 23239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 00:45:56.403020 23239 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580d135f8a044775be2c07c63d72dfee in term 1.
I20260927 00:45:56.403151 23164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580d135f8a044775be2c07c63d72dfee in term 1.
I20260927 00:45:56.403576 23066 leader_election.cc:302] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580d135f8a044775be2c07c63d72dfee, c59afec6f0f048dd899c64916c12e9d8; no voters:
W20260927 00:45:56.409015 23068 tablet.cc:2369] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 00:45:56.409047 23047 master_runner.cc:263] No leader master found from master 580d135f8a044775be2c07c63d72dfee
I20260927 00:45:56.409103 23047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 00:45:56.415709 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.405922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.98.62:40491)
0927 00:45:56.406036 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.406041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.406417 (+ 376us) server_negotiation.cc:407] Connection header received
0927 00:45:56.406639 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.406642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.406689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.406742 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.407781 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.408587 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.409873 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.410234 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.410332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.412164 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.412168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.412170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.412195 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.413898 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.414387 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.414391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.414393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.414475 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.414751 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.414754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.414755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.414976 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.415085 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.415425 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.415546 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 00:45:56.417069 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.417205 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.417279 23300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 00:45:56.417344 23300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [term 1 LEADER]: Becoming Leader. State: Replica: 580d135f8a044775be2c07c63d72dfee, State: Running, Role: LEADER
I20260927 00:45:56.417658 23300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } }
I20260927 00:45:56.419497 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 580d135f8a044775be2c07c63d72dfee. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.419602 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: This master's current role is: LEADER
W20260927 00:45:56.422874 23311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 00:45:56.422950 23311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 00:45:56.424988 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.406500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.0.98.61:41823)
0927 00:45:56.406617 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.406622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.407701 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 00:45:56.407864 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.407867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.407909 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.407955 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.411066 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.411892 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.415021 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.415350 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.415424 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.418006 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.418010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.418014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.418047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.419502 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.421014 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.421021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.421026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.421121 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.423347 (+ 2226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.423351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.423353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.423584 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.423694 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.424678 (+ 984us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.424802 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 00:45:56.425107 23321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 00:45:56.425935 23321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 00:45:56.426774 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:56.407095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.98.60:35217)
0927 00:45:56.407211 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.407215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.409664 (+ 2449us) server_negotiation.cc:407] Connection header received
0927 00:45:56.409825 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.409831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.409881 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.409929 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.412989 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.413813 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.415896 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:56.416293 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:56.416366 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:56.420640 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 00:45:56.420643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 00:45:56.420646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 00:45:56.420672 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 00:45:56.422406 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.422891 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.422895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.422897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.422981 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 00:45:56.423997 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 00:45:56.424001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 00:45:56.424002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 00:45:56.424211 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 00:45:56.424319 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:56.426493 (+ 2174us) server_negotiation.cc:299] Negotiation successful
0927 00:45:56.426612 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 00:45:56.437988 23239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [term 1 FOLLOWER]: Refusing update from remote peer 580d135f8a044775be2c07c63d72dfee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:56.438664 23300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:56.442631 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 580d135f8a044775be2c07c63d72dfee. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.442752 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.445428 23164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [term 1 FOLLOWER]: Refusing update from remote peer 580d135f8a044775be2c07c63d72dfee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 00:45:56.445974 23300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 00:45:56.449416 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 580d135f8a044775be2c07c63d72dfee. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.449528 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.458173 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.458279 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:56.459389 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.459506 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.459833 23323 mvcc.cc:205] Tried to move back new op lower bound from 7333764941562712064 to 7333764941490323456. Current Snapshot: MvccSnapshot[applied={T|T < 7333764941562712064}]
I20260927 00:45:56.461310 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.461416 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee [sys.catalog]: This master's current role is: LEADER
I20260927 00:45:56.463215 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "580d135f8a044775be2c07c63d72dfee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580d135f8a044775be2c07c63d72dfee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "7f3874208d2345e9bafa9df2e110360f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41823 } } peers { permanent_uuid: "c59afec6f0f048dd899c64916c12e9d8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35217 } } }
I20260927 00:45:56.463321 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 00:45:56.464388 23326 mvcc.cc:205] Tried to move back new op lower bound from 7333764941562712064 to 7333764941490323456. Current Snapshot: MvccSnapshot[applied={T|T < 7333764941562712064}]
I20260927 00:45:56.464558 23324 mvcc.cc:205] Tried to move back new op lower bound from 7333764941562712064 to 7333764941490323456. Current Snapshot: MvccSnapshot[applied={T|T < 7333764941562712064}]
I20260927 00:45:56.469084 23321 catalog_manager.cc:1347] Generated new cluster ID: 9109af989f384257baebcc0f05c1a22c
I20260927 00:45:56.469147 23321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 00:45:56.489646 23321 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 00:45:56.490873 23321 catalog_manager.cc:1504] Loading token signing keys...
I20260927 00:45:56.507542 23321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 580d135f8a044775be2c07c63d72dfee: Generated new TSK 0
I20260927 00:45:56.508296 23321 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 00:45:56.540719 23268 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.537609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.98.60:35217)
0927 00:45:56.537733 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.537737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.539652 (+ 1915us) server_negotiation.cc:407] Connection header received
0927 00:45:56.539844 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.539847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.539899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.539951 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.540595 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49544: BlockingRecv error: recv got EOF from 127.0.0.1:49544 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 00:45:56.542552 23119 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.538397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.98.62:40491)
0927 00:45:56.538513 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.538517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.541121 (+ 2604us) server_negotiation.cc:407] Connection header received
0927 00:45:56.541477 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.541481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.541534 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.541589 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.542434 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60796: BlockingRecv error: recv got EOF from 127.0.0.1:60796 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 00:45:56.543617 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.539029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.0.98.61:41823)
0927 00:45:56.539137 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.539142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.541866 (+ 2724us) server_negotiation.cc:407] Connection header received
0927 00:45:56.542890 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.542893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.542942 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.542995 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.543508 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44648: BlockingRecv error: recv got EOF from 127.0.0.1:44648 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 00:45:56.547924 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.546519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.98.61:41823)
0927 00:45:56.546618 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.546622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.547054 (+ 432us) server_negotiation.cc:407] Connection header received
0927 00:45:56.547242 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.547245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.547309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.547369 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.547827 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44650: BlockingRecv error: recv got EOF from 127.0.0.1:44650 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 00:45:56.552631 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.551211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.0.98.61:41823)
0927 00:45:56.551304 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.551308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.551818 (+ 510us) server_negotiation.cc:407] Connection header received
0927 00:45:56.551983 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.551987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.552033 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.552078 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.552547 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44652: BlockingRecv error: recv got EOF from 127.0.0.1:44652 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 00:45:56.557972 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.556388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.98.61:41823)
0927 00:45:56.556478 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.556482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.557110 (+ 628us) server_negotiation.cc:407] Connection header received
0927 00:45:56.557311 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.557314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.557371 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.557421 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.557889 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44654: BlockingRecv error: recv got EOF from 127.0.0.1:44654 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 00:45:56.565364 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.563805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.0.98.61:41823)
0927 00:45:56.563925 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.563929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.564435 (+ 506us) server_negotiation.cc:407] Connection header received
0927 00:45:56.564616 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.564620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.564664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.564713 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.565258 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44658: BlockingRecv error: recv got EOF from 127.0.0.1:44658 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:45:56.577703 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.576104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.0.98.61:41823)
0927 00:45:56.576208 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.576213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.576777 (+ 564us) server_negotiation.cc:407] Connection header received
0927 00:45:56.576996 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.576999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.577046 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.577100 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.577608 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44662: BlockingRecv error: recv got EOF from 127.0.0.1:44662 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 00:45:56.597751 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.596086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44670 (local address 127.0.98.61:41823)
0927 00:45:56.596194 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.596198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.596759 (+ 561us) server_negotiation.cc:407] Connection header received
0927 00:45:56.596994 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.596997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.597049 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.597103 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.597646 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44670: BlockingRecv error: recv got EOF from 127.0.0.1:44670 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 00:45:56.633270 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.631945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44686 (local address 127.0.98.61:41823)
0927 00:45:56.632093 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.632100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.632339 (+ 239us) server_negotiation.cc:407] Connection header received
0927 00:45:56.632501 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.632504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.632550 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.632639 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.633171 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44686: BlockingRecv error: recv got EOF from 127.0.0.1:44686 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 00:45:56.701134 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.699839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.0.98.61:41823)
0927 00:45:56.699981 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.699986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.700137 (+ 151us) server_negotiation.cc:407] Connection header received
0927 00:45:56.700279 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.700282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.700332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.700407 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.701009 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44688: BlockingRecv error: recv got EOF from 127.0.0.1:44688 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 00:45:56.832429 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:56.831202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.0.98.61:41823)
0927 00:45:56.831357 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:56.831362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:56.831511 (+ 149us) server_negotiation.cc:407] Connection header received
0927 00:45:56.831685 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:56.831688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:56.831736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:56.831830 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:56.832308 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44702: BlockingRecv error: recv got EOF from 127.0.0.1:44702 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 00:45:57.091704 23195 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 00:45:57.090565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.0.98.61:41823)
0927 00:45:57.090698 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.090703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.090805 (+ 102us) server_negotiation.cc:407] Connection header received
0927 00:45:57.090978 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.090998 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.091046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.091138 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 00:45:57.091590 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44712: BlockingRecv error: recv got EOF from 127.0.0.1:44712 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 00:45:57.383777 23284 catalog_manager.cc:1259] Loaded cluster ID: 9109af989f384257baebcc0f05c1a22c
I20260927 00:45:57.383903 23284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: loading cluster ID for follower catalog manager: success
I20260927 00:45:57.385573 23284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: acquiring CA information for follower catalog manager: success
I20260927 00:45:57.386740 23284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f3874208d2345e9bafa9df2e110360f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:45:57.403220 23297 catalog_manager.cc:1259] Loaded cluster ID: 9109af989f384257baebcc0f05c1a22c
I20260927 00:45:57.403291 23297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: loading cluster ID for follower catalog manager: success
I20260927 00:45:57.404919 23297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: acquiring CA information for follower catalog manager: success
I20260927 00:45:57.406044 23297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c59afec6f0f048dd899c64916c12e9d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 00:45:57.609939 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:57.605812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.0.98.61:41823)
0927 00:45:57.606052 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.606057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.606205 (+ 148us) server_negotiation.cc:407] Connection header received
0927 00:45:57.606410 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.606413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.606539 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.606657 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:57.606942 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.607463 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.608418 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.608652 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.608759 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:57.609114 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:57.609548 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:57.609686 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:57.609720 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 00:45:57.609772 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260927 00:45:57.615854 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:57.612570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.0.98.60:35217)
0927 00:45:57.612799 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.612803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.612815 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:57.612933 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.612936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.613058 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.613156 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:57.613336 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.613830 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.614578 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.614788 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.614860 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:57.615093 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:57.615457 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:57.615551 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:57.615609 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 00:45:57.615653 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260927 00:45:57.622602 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:57.618913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.0.98.60:35217)
0927 00:45:57.619033 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.619037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.619049 (+ 12us) server_negotiation.cc:407] Connection header received
0927 00:45:57.619087 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.619090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.619129 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.619202 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:57.619631 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.620391 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.621407 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.621727 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.621846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:57.622107 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:57.622168 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:57.622205 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:57.622388 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 00:45:57.622469 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 00:45:57.622860 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:57.618730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.0.98.61:41823)
0927 00:45:57.618853 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.618857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.619195 (+ 338us) server_negotiation.cc:407] Connection header received
0927 00:45:57.619339 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.619343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.619400 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.619450 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:57.619768 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.620252 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.621465 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.621793 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.621880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:57.622320 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:57.622398 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:57.622469 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:57.622658 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 00:45:57.622712 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 00:45:57.623850 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 00:45:57.618437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.0.98.62:40491)
0927 00:45:57.618881 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 00:45:57.618886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 00:45:57.618899 (+ 13us) server_negotiation.cc:407] Connection header received
0927 00:45:57.619014 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 00:45:57.619017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 00:45:57.619162 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 00:45:57.619271 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 00:45:57.619586 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.620409 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.621595 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 00:45:57.621890 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 00:45:57.622013 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 00:45:57.622608 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 00:45:57.622931 (+ 323us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 00:45:57.623043 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 00:45:57.623621 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 00:45:57.623711 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":70,"threads_started":1}
I20260927 00:45:57.626535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 23047
I20260927 00:45:57.632946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 23122
I20260927 00:45:57.638528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskql8uh0/build/debug/bin/kudu with pid 23196
2026-09-27T00:45:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3050 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6143 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279069 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST